Augmented reality smart home system (ARSHS)



The proposed new technology combines the smart home system (SH) with the interactive multimedia and augmented reality display (AR) technology in order to intuitively convey the energy consuption information to user and to fully tap the potential of reducing the energy consumption in a typical residential unit, by changing users' behavior.

Technology Description

Today, due to a steep increase in number and complexity of electrical devices, the control of the living environments (apartments, houses, ...) is becoming an increasingly challenging task for the end user. As a consequence, the typical electrical installations in homes are being substituted with “computerized/automatic” systems, the so called building automation systems, more commonly known as “smart-homes” or “smart buildings”. These systems enable advanced functions while providing easier control and better user experience of processes/tasks involved in a typical household.

A smart home system typically improves at least one of the following areas in the building:

  • Safety & security
  • Energy efficiency
  • User comfort
  • Remote control

In the scope of this technology, inventor will focus on introducing new control technologies to a smart home system (SH) with the aim to fully tap the potential of reducing the energy consumption in a typical residential unit - a household.

A single household is a relatively small consumer of electrical energy. Consequently, energy savings from one household are not relevant in the “big-picture” of energy consumption scheme (of a city for example). Combined however, with more than 25 % of total energy consumption, residential sector is one of the top consumers in urban and rural regions. It is therefore a reasonable assumption that the combined energy savings from multiple households can have a considerable positive impact on the total (urban/regional/national) electric energy consumption.

Typically, smart home systems can be controlled manually either with tactile push-buttons or by using client software, which can be implemented as a web-interface or a native mobile application. Recently, with the rise in popularity of the so-called smart-speakers (Amazon Echo for example), the option of controlling smart homes via voice interface has been introduced. The mobile application and the voice interface provide an added simplicity to the control of the living environment. For the purpose of further improving the user quality of experience of SH systems, the inventor proposes an augmented reality (AR) based control module of the SH system - such a system will make it easier for the user to obtain important energy related information and link this information with needed actions in his/her living environment, which will lead to improved energy efficiency of the household.

Main advantages

- positive impact on the total (urban/regional/national) electric energy consumption

- easier for the user to obtain important energy related information

Stage of development

The technology is in the TRL 3 – Experimental proof of concept.

Currently, a multitude of smart home solutions are commercially available on the market. The ENTIAliving system is a Slovenian solution that will be the basis for the proposed ARSHS system.

Recently APIs for the AR interfaces by Google and Apple have become available. One of these (or similar product) will be used in developing our own AR module. The AR based applications are currently not a mainstream branch on the smart home applications market, but there have been examples with major commercial success in other domains, especially in gaming (for example Pokemon by Nintendo). With major players like Google and Apple coming in to the AR field, there is the potential rise of the ocular projection technology (3D AR/MR HMD smart glasses) that will contribute even further to AR applications becoming a major branch in the mainstream software market, for personal and commercial use.

Challenge and needs

There is an ever-growing need for energy, thus a sustainable and responsible use of energy and resources are needed. Small changes on a larger scale can potentially have an enormous impact. In case of the proposed Augmented Reality Smart Home System (ARSHS) technology, even small increase in energy efficiency and responsible use of resources (i.e. on the scale of individual residential units) can results in large savings on a country scale. To successfully use this technology (user acceptance) and thus positively influence the users’ energy utilization habits, the right presentation (i.e. user interface and interactions) are needed, providing high user experience.

Possible use cases and industrial applications are:

- Active monitoring/control of individual smart homes by their users -> increased sense of being in control of their environment and the ability to make positive changes,

- Gamification of the living environment -> with AR technology it is possible to present the needed actions or relevant energy consumption data in a gaming form. This could further increase the interest of the users, especially the younger generation.

- Smart-home automation system support for making optimal energy saving decisions,

- Easy to use system with state-of-the-art ambient tactile panels and augmented reality presentation on commonly available mobile devices,

- And finally, increased satisfaction and user experience.

Intellectual property

The authors have several patents connected to the proposed technology. The patent (P‐201700311) describes a system and process for automatically controlling the power of one or more cooling systems, and enables short-term control of the consumption of refrigeration systems, where the current electrical consumption can be increased or decreased by specifying the desired reference value.

The patent (SI 23528A) describes a user-friendly biometric interaction method based on simple and intuitive tactile gestures using commonly available 3D accelerometer MEMS devices. The proposed method is important for assuring good usability and user experience, thus increasing the user acceptance of the proposed technology.

Potential markets and targets

According to, the Europe smart home appliances market was valued at US$ 4.08 Bn in 2018 and is expected to grow at a CAGR of 13.9% during the forecast period 2019 – 2027, to generate US$ 12.94 Bn by 2027. Factors such as to rising need for energy-efficient appliances, and demand for comfort coupled with high spending capability of end-users is driving the market growth.

The number of smart speaker users is growing at a compound annual growth rate of 47.9%, according to a new report by eMarketer. That is faster than any tech product since the smartphone, and will result in more smart speaker users than wearable tech users this year. In the U.S. alone, that number will be over 90 million.

Potential partners

Entia Company is a market leader in residential smart home systems in Slovenian market and present in 14 markets worldwide. The ENTIAliving smart home system has been used to equip more than a thousand residential units and more than 200 office spaces. Besides that the ENTIAlivng system has been (or is presently being) installed to automate hotels Mercure, Radisson,…), museums (Museum of Slovenian History), wedding halls and last but not least ski jumping hills (Kandersteg (SUI), Lake Placid (USA), Beijing (CHN),…). Entia Company has access to a multitude of test household SH systems that will be used as a pilot environment.

The HEMS project (NEDO, is a completed project in which Entia d.o.o. has been a partner. In this project 40 houses in Slovenia have been connected to a smart grid in order to control a couple of major loads (in each household) based on a demand from the electricity network. This technology will be used to test the ARSHS system in a demand side management control scheme.

Login to access full technology sheet information