Quantcast
Channel: Infosys-Oracle Blog
Viewing all 561 articles
Browse latest View live

Blockchain in Supply chain & Logistics

0
0

In the past few years, lot of technologies such as Analytics, IOT, Digital, Cloud, Mobility, AI, Blockchain etc. have made life very interesting and we hear about them almost on a daily basis and also about the potential these technologies have. People believe that future belongs to these technologies and they have the power to completely change business dynamics worldwide. It is because of this that organizations, small or large, have started adopting these technologies and their investments have increased sharply in the past few years.

One technology which stands out is Blockchain. Why? It is because while others are still trying to make an impact, blockchain has already taken the lead by showcasing its power through bitcoins- blockchain's first successful product.

So what is Blockchain? Well, it is an unalterable ledger which is shared cryptographically and is used for the purpose of recording transactions' history. Its three pillars are-trust, accountability and transparency.

 What it is not? It is not centralized and thus, no single body or party owns it.

Beyond Bitcoin to Business Blockchain:

We all know that today supply chains are extremely complex in nature and involve multiple players at various stages. Various issues are faced by present day supply chains which can be in terms of lack of transparency, enormous amount of paperwork, lack of interoperability and very less knowledge about a product's lifecycle or its transportation history. All these loopholes severely impact the cost, speed and quality of delivered products.

We can save billions of dollars annually by reducing delays and keeping a check on frauds and according to a WTO report, worldwide GDP can increase by almost 5% and total trade volume by 15% if only we can reduce barriers within the international supply chain.

If we look at the businesses today, almost every business which has the producers, the finance, warehouse management, transportation, regulatory etc. use various ERP systems in order to maintain records.  People differ in their opinions about what is the current state and it leads to disagreements. Not just that, this is an expensive and a highly vulnerable method of doing business. Anybody can alter any record. Here, blockchain comes for help.

 

Some use cases of blockchain in supply chain and logistics:

blockchain use case.PNG

Dry aged beef is a good example. Customers have started demanding organic and local products which have clear origin. They not only want to eat fresh but also want to make sure that the product has its origins as claimed by the beef selling company. How can retailers deal with this?

They can provide product information through apps. Customers can scan the QR code through their smartphones and can themselves validate the authenticity of beef in terms of origin, quality etc. If any discrepancy is found, they can raise their voice. Historical data related to origin (such as feed or breeding), location of farm and of the beef in the entire supply chain, time taken in transport, expiry dates etc. can be easily accessed through a dedicated blockchain database. This is from the view point of customers.

Let's see how can blockchain help an organization. For example: Global retailer Walmart has started using blockchain to track sales of pork meat in China. The system helps it in determining the origin of meat, processing and storage related information, sell by date (expiration date etc.). Also, in case of product recalls, Walmart can understand which batch/batches were returned and who bought them. This way we end up with dynamic demand chains in place of rigid supply chains, resulting in more efficient resource use for all. Isn't interesting? 

Here's an interesting fact about logistics industry worldwide:


market scenario.PNGAbove data, on the one hand, clearly shows the vastness of shipping and logistics industry worldwide but at the same time it also shows the weaknesses of the industry and its inability to deal with such massive rise. In the years to come, logistics market is definitely going to grow but so will be the case with leakages in the system which will ultimately lead to losses. Freight brokers have lot of control over the industry at present. They decide the load, tag on a markup and sell it to carriers. This leads to increase in cost to carriers and ultimately a much higher cost to customers.

Blockchain can help in the sense that it can bring transparency and visibility by allowing shippers to be able to communicate important information such as geo-waypoints, loads and basic compliance information with carriers. Once a shipment is recorded in the blockchain, they can no longer be changed meaning nobody can alter the records or dispute validity of the transaction.

Maersk, the world's largest container-shipping company, is a good example for this. It is working in collaboration with IBM towards building an extremely robust and efficient blockchain network so as to make its logistics arm much more efficient and powerful and with lesser leakages in between. Even some time back, UPS announced the news of it joining BiTA (Blockchain in Transport Alliance). BiTA works in the area of developing blockchain standards for freight and logistics industry.

Blockchain can also be used to maintain history records of trucks. These records can be performance records, maintenance records etc. We all know that once a truck is out in market for sale, buyers wish to know the history of the vehicle. Blockchain with its "unbroken chain of trust" can help.

Drivers can have an important role in blockchain too. They can help in adding their own data, mostly automatic, such as on and off duty, conditions of roads, condition of vehicle and load etc. This can help them in dealing with any kind of disputes with either the shippers or the carriers in dealing with events such as when and where a particular accident happened, goods got damaged etc.

"Capacity monitoring" is yet another area where this can be useful. We all know that cargo volume is one of the main factors which determines the shipping freight charges. Here, blockchain, in association with IoT (Internet of Things), can help. IoT sensors can help in determining the amount of space used by a particular party and can forward this information to the blockchain network. This will help in enabling self-executing payments against the space used by the freight.

Factoring can also be dealt with using blockchain. With the help of smart contracts, blockchain can make factoring less necessary because it will make the complete system of payments towards transactions automatic.

Such cases clearly show that a blockchain can help all parties involved in a shipment to:

  • Reduce or eliminate frauds and errors
  • Improve inventory management
  • Minimize carrier costs
  • Reduce delays from paperwork
  • Reduce waste.
  • Identify issues faster

Keys to Integration successfully

We feel a three-step approach can be adopted to successfully incorporate blockchain into supply chains.

First, the process should begin with an internal blockchain setup which will give sufficient time to the organisation to get used to the technology while making sure that data is available and consistency is maintained throughout.

After this, the blockchain should be extended to other players such as 3PLs and direct suppliers which can help in a robust data management and data exchange.

At last, all the players along the supply chain, including end customers, should be integrated to the blockchain.

If utilized to its full potential, blockchain can improve the customer experience, can drive value end-to-end and eradicate inefficiencies so as to lower costs.

Blockchain Integration into the Supply Chain-A Three-Step Approach

Oval: 2End-to-End

end to end.PNGBY HIMANSHU KHANDELWAL

SENIOR ASSOCIATE CONSULTANT


Untangle spaghetti Model via Order Management Cloud

0
0

There are lot of manufacturing facilities, multiple retail, different finance and procurement centres in different countries, each of these units using myriad custom applications for Supply Chain and each application talks to every application. This is the (in)famous Spaghetti model where the logic on which applications must communicate is hard coded with in each application and this is logic is not configurable. If this sounds familiar, then please read on.

During inception, the organizations chose for one or few application that suite most of their need. But as the organization expands and with mergers and acquisitions, each organization brings its own home grown application. By the time the organization is mature in expansion into a conglomerate, the IT landscape is often a spaghetti of applications.Text Box:  
Picture 1 - Current IT landscape - Spaghetti model with point to point interfaces

The resolution to this situation comes in the form of Order Management Cloud (OMC). The functionality called 'Distributed Order Orchestration' in Order management cloud helps in end to end integration between order entry and fulfilment applications. Below are few key features of OMC.

Interfacing the sales orders: The orders are captured via multiple retain channels like in-store, call centre, ecommerce web site, by engineer during after sales service, mobile application, internal ordering between different entities of the business etc. But these orders can be routed to OMC and created as a Sales order by invoking the seeded order creation web service. The incoming order payload can have different fields populated by order entry system. But as long as mandatory values are present, a sales order can be easily created.

Enriching the sales orders: The SO, so created, may need to have different warehouses where the SO is fulfilled, different booking Business unit based on the geography of the customer, different product needs to be added to the SO based on incoming attributes, can have different shipment method or priority etc. Any transformation on the SO is possible via the pre, product and post transformation rules. To the delight of the IT team, these rules can be built on a visual builder making maintainability of these rules easy

Fulfilment activities made easy: These enriched sales orders are now ready for fulfilment via OMC itself or can be interfaced to different legacy applications for different tasks. For example, manufacturing activity can be fulfilled and interfaced to MES application while a pick and ship can be routed to a WMS application. The invoicing can happen a completely different finance application. All this is possible by configuring the external routing rules and web service connectors for these application. OMC will create a payload of the SO and publish it to these connectors, record the acknowledgement and also the fulfilment of the tasks in legacy applications

Provide complete visibility to customer: As a customer may be curious to know the details of his / her order, OMC can be configured to send a status back at specific intervals. For example, when SO is created in OMC, manufacturing is complete, SO is picked, SO is shipped etc. From IT point of view, this is (again, as you guessed) configurable. The web service connector can be configured for each of the order entry application and OMC will fire the status message to these connectors

Below diagram explains the order orchestration process flow

 

Picture 2 - How Order Orchestration works in Order Management Cloud

Varied business process: The business process may include progressing the sales order via a series of automated and manual steps. For example the SO will have to be automatically reserved, while the customer service team needs to check and update the SO with the customer before the item can be shipped out. Such different processes can be configured via order orchestration in OMC. The SO will be automatically reserved while it will wait for user inputs once the call to customer is made outside the system.

Changing customer needs: In this competitive world, being flexible to changing customer needs is paramount. But at the same time be cost effective. Order management cloud provides functionalities to control the customer change, cost each change and react to each of these changes in a different way suited for the business. The change order functionality can be easily leveraged

Picture 3 - Order orchestration via Order management Cloud

Gone are those days where IT application is just as transaction recording system. IT application is one of the main enabler and enhancer for each business. Order Management, being the revenue making and customer facing module, is truly more flexible to ensure that sales team can be more agile and proactive. So untangle the spaghetti model and route all orders to OMC and dive the fulfilment via simple transformation rules.

Order Management Cloud is implemented as the order routing application in an optical retail chain, operating globally, offering optician services, along with eyeglasses, contact lenses and hearing aids. There are 8000+ stores ordering items via 15+ retail applications and these orders are fulfilled via 10+ different specialised custom applications. With volumes of order line crossing 1 million a month, there is no room for error. While the implementation is still underway, benefits are reaped already by bringing all the routing logic centrally to Order Management cloud.

 

Sathya Narayanan.S

Lead Consultant

Infosys Limited

Bring the power of Excel to Oracle EBS with Web ADI

0
0

Very often we come across business users keeping an Excel sheet beside them and keying in the data into Oracle EBS. If you ask them about automating the process, most often the alternate option that they are  aware of is sharing the file in a predefined csv format to IT support and having them upload it. But what they don't like here is that, any errors will have to be shared with them by IT support and the time and effort involved in the initial upload and the error correction is significant. They feel the effort involved is not really worth it, and they are better off keying this data manually! It is very surprising to see how technology such as Web ADI (Oracle Web Applications Desktop Integrator) is not more commonly used to automate such manual data entry when you already have the data in an Excel or other documents. Most of the time customers hire temps or interns to key in this high volume data into the application.

Without going deep into the architecture of Web ADI, I will put Web ADI as an EBS capability that enables us to create Excel sheets that will automatically connect to EBS and perform the function that it is configured to do. When Web ADI is implemented, the Excel downloaded from EBS will have a new menu for Oracle.

Oracle_Web_ADI_Excel_Menu.jpg

In my experience so far, whenever we have demonstrated the capabilities of Web ADI, the customer response has been enthusiastic . The features that excite them the most -

  • You don't have to login to EBS or navigate/download the Excel every time. You download the Excel once; you can access it by just opening it from desktop. If EBS SSO (Single Sign On) is enabled it automatically understands who you are, if not, a pop from Excel, requests you to login.
  •  You have got all the features of Excel at your disposal - You can drag a cell, copy-paste data, use formulas to derive data, filter for certain values, have your own macros.
  •  You have the option to selectively upload only certain rows from the Excel.
  •  If the solution involves launching a concurrent program in the background, you can monitor the status from the excel itself, without having to login to EBS.
  • Business validations can be built into the Excel to show any issues to the user in the Excel itself. The user can chose to correct it or not upload the error record and proceed with others.
Validation_Error_In_Excel.jpg
  • Can work with the Excel offline. Of course, the user needs to be connected to validate or upload the data
  • The Excel can be configured for List of Values (LoV), double clicking on the Excel cell will open a HTML page that will give the capability  search for valid values and select it.
List_Of_Values.jpg
  • The excel sheet can also be configured to have drop down values.
Drop_Down.jpg
  • When a user wants a report in an Excel periodically, we can have the Web ADI configured for reporting only. The user just needs to open the Excel from desktop and refresh

Web ADI being a module within Oracle EBS, it inherits all the security framework of EBS. When the user is downloading or uploading data from Excel, they are doing it as an EBS user from a valid responsibility. User and responsibility specific restrictions can be implemented. For example, data can only be validated by a user/responsibility and uploaded by a different user/responsibility.

There are of course some areas where Web ADI is not a good option. Web ADI is only for user interface (UI) with one master-child relationship, with one header and optionally multiple child records. If we are looking for data upload with the UI having more than one master-child, it may not be fit for purpose. For example, Web ADI is good to mass update lines for a specific Purchasing Contract, but not fit for a requirement where we need to update the lines, shipments and distributions of a Purchase Order. While these features can be technically implemented, the solution may not be essentially user friendly. Moreover, Web ADI should not be seen as an alternate UI or technology for a Forms or OAF based UI, since it is not possible to dynamically enable/disable columns, show additional info/warning/error message at user key strokes at field level or move control dynamically to different fields in the Excel.

That said, here are the best opportunities where we can use Web ADI effectively

  • Where data needs to created/updated in bulk, where the data may already be there in Excel, etc.
  • Where we need the capability to work with the data offline and sync it up with server when online.
  • Where we may have to review a set of data and correct it - Web ADI can download the data that qualifies the condition into Excel. The user can make the changes and upload back.
  • Existing concurrent programs based solutions where a flat file is shared with the IT support team and the support team uploads it and shares the error records back, on which corrections are made and shared again to the support team for upload.
  • Reports that you want to refresh frequently.

Excel being Excel, most users will be comfortable with it and will not need any special training to use it. Web ADI being a technology module within EBS does not need any additional license. If you already have EBS, you can apply the required patches and start using it. If you look at it the other way, it is a module that the customer has already paid for, and not using it! Identifying the opportunities to implement the Web ADI based Excel solution can be a very good idea, that everyone on Oracle EBS should consider exploring for a better Return on Investment.

Data Quality Overview

0
0
 
  • In this blog we are going to discuss about the Data Quality and to overcome the issues data redundancy, data duplication, Inconsistence data, junk data etc.

     

  • Every organisation are struggling for total data quality - to create and maintain top condition, information that is fit for its intended business purpose. Ente. By working on Data quality justifications are required to filter exceptional data and maximize the trust factor of the Common Data Model. Data quality gives mainly to govern and judge the information.

     

  • Below diagram gives an idea about the data quality flow.

     

 

The architecture diagram below describes a Data Quality analysis as part of Enterprise Data Integration (EDI).

Basically, EDI consists of architectural components such as data sources, Operational Data Store (ODS), data profiler, Data Quality Analysis (DQA), ETL and target database. The target database should be compliant with the industry standard SID framework. A SID-based  CDM is a catalyst in the integration architecture since it enables CSPs to integrate disparate data sources and data from mergers and acquisitions with minimal rework.

 

Data Profiling

 

Data profiling will inform about the state of the data and based on the data state we have to create rules and apply on the data.

In below table we can see some discrepancy with the data which are highlighted in blue, red and green colors are depicts classic examples of data anomalies.

Data profiling also helps to discover data problems such as inconsistency, missing data, duplicate data, data that does not meet business rules and orphaned data.

Name

Age

DOB

Gender

Height

Anomalies

Bob

35

13-1-81

M

6

Nil

Rob

34

27-8-82

5.5

Lexis Error

Madan

34

15-1-82

M

5-9-2

Domain Format Error

Diana

33

Duplicates

Jim

0

12-7-88

M

5.1

Integrity Constrait Violation

~@$

^^

@

#

Missing Tuple

 

Data profiler supports data analysts to find metadata information such as data type, decimal places, data length, cardinality, primary key, unique count, unique percent, pattern count, minimum value, maximum value, minimum length, maximum length, null counts, blank counts, etc. Such analysis helps determine whether the data adheres to metadata definitions and the business expectations.

The assistances of leveraging data profiling include:

  Data as per business expectations

  Determining the level of quality of each attribute

  Deciding the type of cleansing, standardization or de-duplication required to meet business expectations

Outline matching is an advanced analysis on any given attribute whose distinct value is very low. Below table shows the result of a typical pattern matching analysis on the phone number attribute. It helps a subject matter expert to frame Data Quality rule on phone numbers.

Outline

No. of Rows

Percentage

222222

1456

80

21365-Aaa-1542

3498

9.5

[2345]1234-999

1000

12.1

7777-123-7777

1500

9.8

6666-aaa

45

0.1

 

Here are few examples of Data Quality rules.

  • Phone numbers containing Text should be stripped and should contain only numbers.

  • Gender should be either M or F, and any other entries should be defaulted to blank.

  • For records having name as null, the records should not be loaded into the Database.

 

Data which is having poor quality, incomplete and inaccurate ca over come by doing data auditing.

A perfect data reviews detect key quality metrics, missing data, incorrect values, duplicate records, and inconsistencies. When used in combination with Oracle Enterprise Data Quality Parsing and Standardization, it can deliver unique understanding of your data. 

Results of these profiling and audit processes are presented in easy-to-understand executive dashboards. By usig Data quality dashboards allow problems to be quickly identified, before they start to cause significant business impact. Graphical views show data quality trends over time, helping your organization protect its investment in data quality by giving visibility to the right people.

 

In this part we will discuss how to cleanse the data. Data cleansing task performs to remove duplicates, anomalies, inaccurate data from the source.

Data which has passed quality testing and can be passed for ETL processing.

Data cleansing is the method of locating and repairing data anomalies by comparing the domain of values within the database. Data is scrubbed or cleansed as per the corrective measures suggested for exception like misspellings, missing information, invalid data, varying value representations, etc. Anomalies are fixed automatically by ETL processing.

The DQ and ETL framework must understand, interpret and re-organize data automatically within any context, from any source, to the standard target form. 

 

Data Quality engine finds that any data item "stands-out" (holds statistically significant variance from a mean population), then the engine flags it as an exception and stores it in the exception schema. Such exceptions are thoroughly analyzed by the data governance team or subject matter experts. Depending on the category, exceptions are communicated to:

1.  Data factors to fix data anomalies at the source database.

2.  Quality experts and business users to frame new quality rules/corrective measures.

3.  Exceptions captured during data auditing and ETL processing are analyzed to generate scorecards and dashboard reports.

4.  The data quality framework automatically measures, logs, collects, communicates and presents the results to those entrusted with data stewardship and the source data owners

 

Quality Dimension

Accuracy

Uniqueness

Integrity

Consistency

Density

Completeness

Validity

Schema Comformance

Uniformity

Vocabulary Errors

Format Error

Irregularities

Missing Value

 

Indicates Direct downgrading of the quality dimension

 

Indicates that the occurrence of this anomaly hampers the detectio of other aomalies downgradig the quality dimension

 

Chatting with Bots - More necessity than a science fiction

0
0

In the age where there are multiple applications involved in supply chain process, the knowledge about the customer orders is distributed. It has become a walk on the tight rope to keep the customer updated about the process of their order Vs cost to provide the information to the customer via customer service team or a complex BI solution. This blog opens a possibility of cost effective and light weight solution by introducing the 'Chatbot'.

The IT landscape involve multiple applications to fulfil every single order due to the nature of business, way the organization have evolved, number of business entities involved or due to the speciality of the applications. Below is the example of a manufacturing and retail organization

Pic 1 - Typical IT landscape

In this complex matrix, the traditional methods to keep the customer updated about the progress of their orders are as follows

  • Send text message or email about the status
  • Set up a customer service team to handle customer requests via call, text, email or chat

But the drawback of these conventional methods are that there is no single system which holds the moment of truth about every order. In order to avoid the customer service team juggling between applications, a complex BI reports are installed to oversee all applications resulting in even more complex IT landscape.

Alternative solution is that 'Chatbot'. According to Wikipedia, a chatbot is a computer program which conducts a conversation via auditory or textual methods. Customers can chat with Chatbot to get the information about their orders. Let's see why the Chatbot solution is cool.

Implementing the Chatbot:

PIC2.png

There are 2 main functionalities of Chatbots:

  • Receive and understand what the customer is saying, and
  • Retrieve the Customer information required

 In order to receive and understanding what customer is saying via chat, Chatbot uses Natural Language processing systems. Via artificial intelligence and machine learning, Chatbot is trained to understand the customer's request better. There are numerous cloud based chatbot development platforms can be leveraged to design, build and train the Chatbots. Oracle Cloud Platform or IBM Watson are examples of such Platform as a Service (PAAS)  solutions available.         


Text Box:  
Pic 3 - Example of a chat conversation in mobile
For retrieving the information required, the Chatbot uses web services to connect with each application. For example Order management Cloud has an Order Import Web service which can be involved by using the retail order number. Similar order information web service can be created. The Chatbot will have to invoke the web service and find out the best status of all the application and publish it to the customer.

Via these NLP and web services, implementing a Chatbot solution is easier than ever.

These Chatbots are not too bulky and intrusive like traditional BI solutions. They occupy less space in server or can be easily placed in Cloud as well.

Customer Experience:

Customer Experience, in short CX, is a major focus area for the organizations. With referral customers giving more business than new customers, the organization want the customer to be handled with care. The Chatbot will give the customers an unparalleled experience just like chatting with a human.

The Chatbot can chat in different language as preferred by the customer. In addition, Chatbot can be trained to reply on text or voice commends as well.

The Chatbot can be used on computer, tab or even mobile to give customer an excellent convenience.

Capex, What Capex?

 Setting up a multi-language enabled customer service team 24 x 7 or implementing a complex BI solution is far more costly for the organization. The cost and time to implement a Chatbot is far less when compared to the traditional methods. Readymade Chatbots are available which are already designed and built to a general extend. The implementation will be limited to involve the order information web services from various application and to train the Chatbots.

capex.png

The Chatbots can also be used for expediting an order if customer requires. Chatbot can send mails to the Production team in manufacturing facility with the chat history to ensure that the order is expedited.

With the technical advancements, Chatbots are even helping patients who suffer from Alzheimer's disease and insomnia.

To summarize, Chatbots are easy, simple and light weight applications that solve the major problem of keeping the customer engaged. So if you are chatting on a web site to know the status of your order, you may be chatting with a robot already!!!

OBIEE vs Tableau: Competitors or Complimentary?

0
0
 

OBIEE vs Tableau: Competitors or Complimentary?

Introduction

It all started around 5 years back for me when I started to explore OBIEE for reporting.  Whole expertise that I had by then in OBIEE and I wasn't introduced to any BI reporting tool till then

10 months back, when for one of the client I was working for decided to opt for Tableau 10.3 with the assumption that they will start any new developments in Tableau even though OBIEE will remain live as it is, but gradually all the existing reports would be slowly migrated to Tableau, moving away from OBIEE (even though they will retain all the data model that is setup for Datawarehose that already exists).  We delivered pilot phase for this and this is still in progress but this made me interested to know (and ultimately train myself in Tableau) about what exactly this new reporting tool is, and I started to explore it via various learning modules available online as well as on COE of client. 

Initial Thought

I was under the impression that it would be just like BI Publisher and just to make reporting easier (which is also one of the goal), client wants to implement it over OBIEE.  But that didn't turn out to be the case. 

Pros and Cons

I won't go into technical details here but would like to do a pro versus cons analysis of OBIEE and Tableau from point of view of client, developer as well as a reporting user

OBIEE

Tableau

Approximately 27 connectors including ODBC connectors.

70+ supported data sources via in built connectors along with Tableau Support.  Even more using ODBC connectors

Not easy to use for an organization or business users even if a minor change is needed immediately.  They will have to go via IT team

Once data model is ready, minor changes and even new derived columns can be easily published online within seconds

Any new table or data source can be connected as and when needed.

Not each and every data source can be brought into data model easily.  Custom SQL can be used but can't be tuned for performance improvement as Tableau server doesn't handle SQL performance of custom SQls

Has strong component of BMM layer and is a unique feature of this tool

All the circular joins should be handled in the physical data model only.  No scope (apart from data blending at the cost of ) of joins at the later stage other than data model

Map viewer is difficult to implement and manage.  Moreover needs a separate setup

Has inbuilt visualization of Maps with one click implementation.  Has ability to draw custom maps as well using geo spatial data

Has a few analytics functions viz. PERIODROLLING, AGO etc. but they do come at the cost of performance degradation.

Has very efficient Window functions that can help to use derived columns as a data source at run time of visualization

Oracle is now ending support of non-Oracle ETL tools which is a kind of restriction

Any ETL tool can be used

Out of the Box are valuable if you want to use one of the provided data model

No Out of the Box data model is provided

Total License Cost for Enterprise Edition is $51800 without any Support and user access (based on latest cost sheet from Oracle)

Cost for professional edition including Client and Server is $1260  per user per year including support(with yearly billing)

Product crashes with not so sure error messages

Quite stable tool and very rare issues.  Has very detailed error and exception handling

Can handle and manager large business models very easily

Large data models should be kept offline manually unless published online

No predictive analytics

Has inbuilt capability of predictive analytics and integration to R language

 

As it is quite clear about the advantages of Tableau over OBIEE, we can conclude the reason why Tableau took over OBIEE in last 5 consecutive years as a leader.

Gartner.png

Image Source: https://optimalbi.com/blog/2017/02/17/gartner-magic-quadrant-for-business-intelligence-2017-cloud-is-coming-slowly/

 

Conclusion

Even though both the tools are being compared, but if a new organization is about to opt for a BI tool, it should start from Tableau and can outline its BI needs.  If it feels that business needs are too large to be handled by Tableau, it can bring in OBIEE as well.  Both the tools can co-exist as well and would be a great advantage to the organization.  Business users do feel Tableau to be an easy to use tool (where ad hoc reporting can be done very easily using offline files as well).

However Tableau has an added advantage of Predictive Analysis and can connect to powerful R Studio that isn't even a feature in OBIEE.  Oracle has it but as a part of separate predictive analysis tool which is a costly affair.

We can't say that both the tools are competitors but do complement each other in many ways.  While we don't need too much of technical know-how about Tableau architecture to work on it, OBIEE does need support from IT team that is well aware about the architecture which makes Tableau a better solution where most of the users are not too familiar to use OBIEE.

SAS Model Risk Management

0
0

In this blog, will discuss about model risk and how model risk management is important for business needs and how beneficial by implementing the solution using SAS.

Here we will majorly focus in the finance sector and refer in terms of Financial Services business needs.

Decision making plays a very important role in any organization's business. In order to make the efficient decision making, building model which captures the specifications of these decisions is very crucial.

Hence the loss which is resulted by using inappropriate or inefficient model is nothing but Model Risk.

So nowadays Model Risk Management has captured plenty of attention among various risk managements for many of the financial institutions.

In order to mitigate the model risk, majority of financial institutions are in the process of developing Model Risk Management frameworks.

Not only financial institutions, but also the regulatory authorities are also keeping a tab on these frameworks by enforcing certain standards in order to bring model risk awareness to effectively identify and manage model risk along with governance of entire model life cycle.

Challenges in Model Risk Management:

1) Governance: In order to comply with the standards enforced by the regulatory authorities, the financial institutions should maintain a centralized repository which contains all the required documentation, reports, checklists, dashboards, codes and regulatory feedback.

2) Data Management: Financial institutions are supposed to submit various documents/reports/dashboards to the regulatory authorities at regular intervals as per the policy. In order to submit those documents, institutions has to collect and store historical data related to customers and facilities. Not only collecting and storing the data, they have to ensure the data quality which will comply with the standards as per regulatory authorities.

3) Monitoring and Validation: Last but not the least, monitoring and validation is very crucial in the model risk management. Ongoing monitoring against the market at periodical intervals is very important for any business. In order to avoid letting small issues snowball into larger issues later on, model validation at various stages by performing various model and factor level tests which capture stability, trend and robustness of the model is essential to identify the accurate risk.

 

Benefits of implementing solution using SAS to mitigate challenges of Model Risk Management:

1) Centralized model repository: Irrespective of the model type, technology, platform where the data resides, SAS will allow you to access the information thereby providing the end to end workflow management.

2) Data management: SAS allows you to collect and store the required historical data accurately and also makes sure that it is aggregated by identifying the relevant fields for the creation of the data marts or repository which will connect with the live systems offline and store data. This will effectively govern the entire data management for the model risk management.

3) Model monitoring and validation: SAS will enforce the appropriate testing on the platform for various model and factor levels along with adequate reviews in order to adapt the regulatory guidance on monitoring and validation.

 

 

Binding and its functions in Einstein Analytics

0
0

In this blog I will cover binding in Wave Analytics with a brief introduction on Wave Analytics.

This is an analytics tool which is developed for Salesforce and it is a cloud based analytics which connect data from different sources. It creates interactive views and shares the views in Apps. It is very useful for the business users who can check the changing trends in the market and take necessary actions.

Binding feature enables interaction between the reports in the dashboard as a prompt/filter in OBIEE. There are two types of bindings in wave analytics namely selection binding and result binding. By using these we can select different selection types by which the reports in dashboard changes as per the selection or by using result binding we can change the result of one step by triggering the other.

Before moving deep in to this we will discuss about faceting. By here we understood that the binding is used to create interactions between the widgets or simply the reports in a dashboard in wave. Faceting is simple and easy way to implement the interaction between widgets. It is an inbuilt feature in wave analytics.

We have an option in steps where we need to tick the checkbox, where it says "Apply filters from faceting". But why we go with the binding is, there is a limitation in faceting that we can't use this option with the dashboard contains lens(reports) from different datasets. So we go for binding.

PFB the screen shot showing the faceting check box for a widget:

 

 

 

 

 

There are two types of Binding, one is Selection and other is the result binding:

1.      Selection Binding:

The interaction in one step effects the other step, then it is known as selection binding.

Example:

In the below screen shot the two bar charts are Sum of Sales by Owner and Sum of Targets by Owner.

After applying the selection binding, once we click on any bar in the dashboard the other chart changes as per the owner selection. Here the selection binding is on the owner column.

 

 

2.      Result Binding:

If a step changes based on the result of another step, then that type of binding is known as result binding.

Example:

The below screen shows the result binding, here the bar chart changes as per the column in the toggle. By clicking on the top 5 button in the toggle we are resulting the bar graph to show top 5 average of amount by Account Industry.

 

 

 

Binding Functions:

We have three types of binding functions. They are:

v  Data Selection Functions,

v  Data Manipulation Functions, and

v  Data Serialization Functions.

By Using Binding functions, we can get the data from the step, manipulate it, serialize it and make it used by the target step. These functions can be used on data like scalars (0, 'This is Einstein analytics' or null), one and two dimensional arrays.

Binding functions are nested functions and must contain one data selection function, one serialization function and any number of manipulation functions.


For Example:

{{column(step_1.selection,[\"Owner\"]).asObject()}}

Here in the selection binding example when we click on any Owner in the first chart the second chart changes. As we send the Owner column as an object, when a particular owner is selected the dashboard changes as per that.

 

1.       Data Selection Function:

 

The source data is selected by selection function. It can either be a selection or result of a step which returns table of data. In which columns should have names and rows must contain an index (starts with 0). We can select columns or rows or a cell for binding.

When one row and one column is selected, it returns one dimensional array or if it selects multiple rows and columns then it returns two-dimensional array or if a cell is selected, then it returns a scalar value.

 

We have three types of selection functions, they are Cell Function, Row Function:

 

a.       Column Function. Cell Function: Returns a single cell of scalar data, where row index should be an integer and a column name should be a string and the cell should exist.

{{cell(step_1.selection,[\"Id\"]).asObject()}}

 

b.      Row Function: Returns one row of data as one dimensional array, or multiple rows which as two dimensional array.

{{row(step_1.selection,[\"Manager\"]).asObject()}}

 

c.       Column Function: Returns one column of data as one dimensional array, or multiple columns which as two dimensional array.

{{column(step_1.selection,[\"Owner\"]).asObject()}}

 

 

2.       Data Manipulation Function:

This function manipulates the data which is required as per the data serialization function. This function can be applied on the results of a selection function or even on the result of another manipulation function. The input can be null, if so, it will return a null value.

If there is no requirement for the data manipulation function, we can directly add a serialization function to the results of data selection function.

The data manipulation functions are as follows:

1.       Coalesce function: It is used to provide a default value when null value is returned.

2.       Concat Function: Concates data from multiple sources and returns as a one or two dimensional array.

3.       Flatten Function: It flattens a 2 dimensional array to a single dimensional array.

4.       Join Function: Converts one or two dimensional array into a string.

5.       Slice Function: From one dimensional array gives first and optionally the end position and returns the one dimensional array. It supports negative indices.

6.       toArray Funtion: This function basically converts the data to array, for example if the data is in scalar form it changes to one dimensional array or if it is in one dimensional array then it changes to a two dimensional array.

7.       valueAt Function: As the name suggests, it gives the value of a particular index which is requested for.

 

3.       Data Serialization Functions: This function converts the data as per the binding requirement.

The below are the different types of serialization functions:

1.       asDateRange(): this function returns date range filter..

2.       asEquality(): this function returns 'equals to' or 'contains in' filter.

3.       asGrouping(): this function is used to return a single or multiple groups.

4.       asObject(): this function returns data as object.

5.       asOrder(): this function returns sorting order.

6.       asProjection(): this function is used to project a field in step.

7.       asRange(): This function returns a range filter.

8.       asString(): this function returns a string.



Managing big teams :24X7 support

0
0


Dear Friends, 


There is a famous saying 'Teams of ordinary people achieve extraordinary results'. This saying holds true in achieving the results always, especially, when it comes to managing big teams. The first and the foremost strength of the team is communication. If communication breaks, then, things start falling apart. From disrupted client deliverable, to the bad rapport amongst the team-members. Even the biggest of the space missions fail would at certain points, when people stop talking. 


Few key points in handling big teams


There might be number of challenges in handling the teams, when they are operative in 24 X 7 kind of model, but, everything is achievable if some thinking is put into it. Here are few corrective and constructive measures to be applied.


Meetings:  Daily meetings must be organized to sum-up the day's activities which were handled by different shifts. What-went-well and what-went-wrong should be discussed in depth. But care should be taken that the meetings should not eat a productive part of the day. People have best energy during the morning hours or start of the shift. 


Hand-Overs: Shift hand-overs should be properly done. The progress can be shared with the entire team via emails. PMs and Leads should define a process of the shift-hand over. Well-defined shift-handover emails / progress emails should be shared by the shift leads with entire team. A properly delegated task works wonders in achieving the desired results.


Technical Leadership: Team should be built strategically with correct number of technical leadership to support appropriately, in times of distress. Improper leadership can deface the name of an organization easily.


Resource Planning:  Resources should be hand-picked, so that head-count be divided wisely across all the shifts, without impacting the resource availability and work. Resources with versatile skills should be equally distributed. It's advisable to divide the team in small groups for better output.


Skill Planning/Productivity Check: It's advisable to get monthly skill check done for all the candidates working in the team. This ensures that team is well equipped based on the need of the hour, changing technologies, client commitments and expectations.


Project Management Activities: A PM carries the responsibility on the shoulders to build the team, maintain CMMI accreditation of the project etc. Quality assurance, resource planning, escalations are to be handled by the PM.


Small Steps Towards Milestone: There should be realistic and achievable milestones. Check-points for milestones, play an important role in keeping the things moving. This also ensures that motivation level of the team is up and doesn't fall apart. Client should be aware of the check-points and should be in agreement with the same. They should be kept informed in a timely basis.


Communication and proper planning is the key factor for better team management. Even "Impossible" says "I-M-POSSIBLE". Most impossible tasks will become possible if proper steps are taken. Hope this article helped you plan your teams better. Let us know in the comments .




Article 0

0
0
 

This is part 2 of 3 blog series comparing PCMCS - Profitability and Cost Management Cloud service Application; and HPCM - Hyperion Profitability and Cost Management on premise.

In the last blog I had written about what is Oracle Profitability application and the areas in which it can help an Organization. Also the modules that are part of HPCM but are not present in PCMCS. So in spite of having only Management Ledger module functionality why PCMCS is still the way to go.

For one, all future roadmaps from Oracle talks about development in cloud modules. The last big feature development in HPCM on premise was in year 2014. The focus from Oracle is to enhance features in Oracle EPM Cloud and associated modules. Be it in terms of tighter integrations from data perspective with Oracle source systems or metadata integration with DRM in cloud.

Apart from thrust from Oracle to move customers towards PCMCS, there are many other reasons that makes PCMCS a better bet than HPCM, such as data management, automations, Dashboards etc. In this part of blog series let's look at below 4 areas in detail:

1.     Data Management / Integrations

2.     License Cost

3.     Future ROADMAP

4.     Available Automations

Data Management:

To connect and load data from source, PCMCS uses data management interface (limited FDMEE). FDMEE is an ETL tool wrapper used specifically for many of the EPM Products. In HPCM the customer would need to procure an ETL tool license in addition to HPCM license whereas PCMCS license cost includes Data Management within same cost. Data load, error reporting, load rule scheduling and its automations can all be managed efficiently using Data management window. Yes it's still file based load without any direct adaptors with premise source systems, but the fact that build and maintenance of data interfaces is from a single user interface makes it a good solution. Few reasons why it works for customer are:

  1. License cost of an ETL/ELT: Data management is included in PCMCS license cost, so customer need not buy an on premise ETL tool to connect to various source systems.

  2. Build and maintenance of data interfaces is from single user interface: data management interface is accessible via same PCMCS workspace. So for any mapping change, profile update admin need not access and connect to another ETL UI.

  3. Ease of maintenance: once users are trained on data management, they could edit and maintain interface with ease. The users are not required to upgrade/maintain/secure a separate ETL source.

  4. Out of box features of FDMEE such as error reporting, workbench for reconciliation are all available within data management.

License Cost:

License cost for HPCM is calculated based on Per Million Revenue of an Organization. PCMCS calculation follows per user based licensing. As HPCM / PCMCS is an allocation tool to be used by specific Finance users who are typically not very high in number. So for an Organization with very high revenue the License cost of HPCM could become a deterrent but PCMCS could easily fit within the budget of all Medium and high revenue Organizations.

Future Roadmap:

 

Automation:

Any automation of jobs in PCMCS is made simple due to availability of EPM Automate utility. It is one unique and good feature available with Cloud EPM, which enables automation of several regular tasks in PCMCS including but not limited to following:

  • Backup of metadata, data and application etc.

  • Refresh and Deploy of application

  • Generating archival files, storing them in server location and deleting files older than specified date

  • Manage POV in PCMCS (copy, delete etc.)

  • Executing data management rules

  • Uploading and deleing files into/from service instances etc.

     

    So in a Nutshell if we have to compare the effort and Cost difference for a HPCM or PCMCS implementation there would be an extra effort of 8 to 12 Weeks for HPCM implementation:

Activity

Cost & Effort

Hardware sizing

HPCM Installation

Required activity.

Around 2-3 weeks effort is required to perform hardware sizing, Server configuration, error resolution and HPCM installation.

Not required. Server is provided and maintained by Oracle so no hardware configuration activity required.

 

FDMEE Installation

Required activity. Around 1 week of effort

Not required as data management is already part of PCMCS

Application Creation

Requires same effort.

Requires same effort.

Allocation build

Requires same effort.

Requires same effort.

Data Interface build

Data Interface build in via separate ETL Window. Any change will need user to login to this window to create/edit mappings. Maintenance of this ETL is also separate activity. 2 Weeks of effort

Data management is part of PCMCS navigation window. Any interface creation is via PCMCS dashboard and maintenance is included in regular application maintenance window

HFR /Smart view Report Build

Requires same effort.

Requires same effort.

Dashboard Build/availability

Dashboards are not available

Prebuilt dashboards are part of PCMCS Configurations

Automation Job

No standard automation tool available. Custom jobs need to be created for automation and would require maintenance as well. Requires 2 weeks extra effort.

EPM Automate is part of PCMCS License, hence automation job scripting is easier to create and manage

Application Maintenance

Maintenance custom jobs need to be created. 1 Week effort

Daily Maintenance of complete application is part of Oracle Maintenance schedule.

Disaster recovery

Disaster recovery from backup plan needs to created and backup maintained as required. 1-2 weeks extra effort

Disaster recovery is by oracle team

Total Extra Effort

8 to 12 weeks

 

 

 

OBIEE Integration with Tableau

0
0

         In this  blog I will discuss and show how Tableau - OBIEE integration can be achieved and  options to connect to several data sources from Tableau.  It's a frequent ask from clients where OBIEE and Tableau present in reporting area.

OBIEE and Tableau are known as Business Intelligence Reporting tools. It is possible from Tableau to connect to OBIEE metadata file, build reports and provide advance data visualizations using Tableau.

What it Needs?

OBIEE Client tool installation in Tableau Desktop System

OBIEE application details to connect from Tableau

How to achieve?

  • Create a ODBC DSN in Tableau desktop system using driver: Oracle BI server.  Provide OBIEE server name and port details and test Cluster connectivity

Hari1.jpg

  • Create a Tableau Data Source connector(Tableau.tdc) file with the specifics as provided below

HariTdc.jpgReporting from Tableau by using ODBC Data Source

Tableau allows to use multiple data sources for a single worksheet. The below provided are the section steps for connecting to ODBC data source from Tableau; to be followed creating ODBC DSN.

Step1: Launch Tableau Desktop

Step2: In Tableau Desktop, click on Other Databases (ODBC)

Hari2.jpg

Step3: In DSN drop down,you will find the DSN you already created. Select the DSN you just created. Click Connect

Step4: There will be a popup screen appear. In Login window, provide credentials to connect to. Click OK Button

Hari3.jpg

Step5: On providing correct credentials, It shows window with message: Connection has been established successfully with ODBC Data source

Step6: You can create data model in Tableau Book by selecting the required OBIEE Presentation Tables that appear on left navigator. And you continue to build the report

Reporting from Tableau using Multiple Data sources

Tableau allows to use multiple data sources for a single worksheet. One data source is primary; all others are secondary. 

The below are the Key Data sources:

  • File-based data sources

  • Relational data sources

  • OLAP data sources

  • Cloud-based data sources

    In this blog, I'm discussing how we can use File/Relations data source in Tableau

  1. Creating a Data source to connect to a File: 

    The below steps are to be followed to connect Tableau to a file data and set up the data source.  The below steps help to establish a connection to text file. 

  • Launch Tableau Desktop

    Hari4.jpg

  • As shown in the below screenshot, select Text File to connect to a text data file,  In case you want to connect to Excel file, select Other Option : Microsoft Excel

    Hari5.jpg

  • In the next screen, select the file path from network

    Hari6.jpg
  • In the next screen, select the Connection Type: Live /Extract
  • When Live connection is selected, a direct connection to the given file path will be established

    Then select Goto Worksheet to go to the work book to do Reporting by selecting the fields from the connected file. Any update in the source file reflects in the Reports accessing this Live connection.

Hari7.jpg

      When Extract Connection is selected, it prompts to save the file in a local location, So that the extracted data will be stored in .tde file.

    The above .tde file can be used in any Workbook to work on the data available in the local Tableau data extract.  This step needs be repeated if there is any data refresh in the source file.

2.    Data Source creation to connect to a Database Table:

The below steps are to be performed to create a data source to connect to database tables. The given example illustrates steps to connect to Oracle Database.

  • Launch Tableau Desktop

  • Select Oracle to create Oracle Database connection

Hari9.jpg

  • Provide the Connection details to connect to the Oracle database

Hari10.jpg

  • Click connect, Select the Schema Name, Table Names. When Multiple Tables are selected, provide the join among the Tables.  Click Go to Worksheet, where the Report can be built using the fields selected out of the Tables 

    Hari11.jpg

Data Blending using Multiple Data Sources

Tableau features to connect to multiple data sources to enable Data Blending for combining data from multiple data source types in a single worksheet.

Once multiple data sources are connected in worksheet, the Relationship between the data sources has to be defined on the Joining columns as shown below.

Hari12.jpg

Once the Relationship is defined between data sources on joining columns, the Report can be created by dragging the fields from the defined data sources as shown in the below screenshot.

Fields highlighted with Blue border are dragged from a Text File Data source.

Fields highlighted in Green border are dragged from a ODBC data source created to fetch the data from OBIEE Subject Area.

Hari13.jpg

Reuse of Data Source Creation:

Once a Data source of any type( File/Excel/Database Table/ODBC ) is created in Tableau, this Data source can be saved by using the option : Add to Saved Data Sources in Tableau as given below. This Data source can be reused in other workbooks if you want to reuse the same data source (same file/Table/model) again.

Hari14.jpg

Save the Data source in a location. This Data source can be opened in other workbooks so that this can be reused.

Hari15.jpg

Caching

  • OBIEE "nqquery.log"/OBIEE Cache doesn't capture the log for the reports you create from Tableau

  • Caching (Browser Cache) can be managed at Tableau level as all the reporting happens on Tableau

Security

  • OBIEE and Tableau must have a common Authentication mechanism to pass the user credentials from OBIEE to Tableau (enabling Single Sign On).

  • Authentication is done at OBIEE. Data Level Security is managed/controlled within Tableau.

Key Features:

  • Amazing Data Visualization

  • Direct Database connection and report creation

  • Easy Dashboard Publishing and sharing

  • All general configurations like deriving calculative fields, report drill capabilities, hierarchies supported

  • Dashboard Export functionality for the selected worksheets

Key Considerations & Limitations:

  • OBIEE is not a trusted data source in Tableau Desktop. As a result, when using the generic ODBC data connection with an unsupported data source, success often varies and full compatibility with Tableau features cannot be guaranteed.

  • All the tables across OBIEE subject areas presented to the user on Tableau data sources tab.

  •Its recommended to expose the foreign keys in OBIEE RPD in Presentation layer in order to frame the joins between the selected presentation tables in Tableau. 

  • Select the subject area for which ODBC connection needs to be created.

   All the tables across OBIEE subject areas presented to the user on Tableau data sources tab. Key attributes of all the Dimension tables need to be exposed in the Fact Table in OBIEE RPD to configure joins in Tableau Data Sources. At the same time, Exposing Foreign Keys in the Presentation layer is not an OBIEE best practice. It's suggested to restrict the foreign keys in OBIEE RPD to Administrator, so that the Keys gets exposed in OBIEE RPD for usage in Tableau and these will not be exposed to OBIEE end user/Business users while performing adhoc reports.

  • Data Sources from External source (Files/Tables) can be saved in Tableau, so that it can be reused if the same data model is needed in the other workbook. In case it needs a data model other than that present in the saved data sources, then new connection has to be recreated every time.

Robots - Their World, My World, Our World

0
0

Everyone is talking about, and to, Robots

Robotics, a buzzword that is heard almost anywhere and everywhere now. A concept, that was once upon a time limited to hi-tech manufacturing industries, has now spread leaps and bounds to almost every place or application one can imagine. These days, you don't have to scout through pages of a technology journal or thought paper to learn about innovation in the field of robotics. The subject has become so popular (and rightfully so) that it has found a place even in everyday articles, newspapers, television shows and so on. As per a recent survey, Robotics is one of the top 5 trending technologies sharing this prime space with Artificial Intelligence, Internet of Things, Big Data and the most recent entrant - Blockchain. I personally think that the primary reason for this stardom, of course apart from all the 'cool' things robots can do, is how close they have got in our regular routine. In today's modern world, we are always interacting with robots, sometimes knowingly and at other occasions, obliviously. What's more interesting is the fact that this interaction is not just limited to the period when we are awake and active. Even if we are sleeping, some of the robots, that are part of our day-to-day technology savvy life, aren't. For example, smartphones, smartwatches and those little bracelets called fitbits keep working while we are in deep slumber keeping a track of sleep patterns, REMs, heart rate, pulse and much more. But isn't this what they are supposed to do?! For some people (a.k.a humans), the bigger question could be "why", for others it might be "why not". And then there is also another faction who would ask "how?". This article is intended for all of the above.

 

Evolution

Back in the day, the purpose of a robot was to obey commands and do the job assigned. It was a one-way communication and the response (implied, not explicit) from a robot invariably had to be, as they say in military language, "Wilco" (Will comply, with the orders). Over the years, even though the primary purpose has remained the same, robots are doing much more than just performing a set of mundane tasks. In fact, even our expectations from them have grown exponentially and the robots, fueled with new technology, are coming through with flying colors.

Robots have evolved from something having a form, fit and function to something that can work equally well, or maybe even better, with just a fit and function but no form. In simple terms, not all robots can be "seen". We refer to this category in our IT world with a term or jargon - Soft Robot or just "bot". Here are some of the key differences between that and a typical industrial robot. By no means one can be deemed superior than the other. It's the requirement and application which makes either one just more relevant or practical to use.

Characteristic

Physical Industrial Robot

Soft Robot

Place of installation

Fixed

Flexible

Dimensions

Wide range, depending on the actual equipment

Dimensionless (virtual)

Type of job performed

Generally same but also set of relatively similar tasks at times

Any - even if one task is strikingly different than the other

Availability

Mostly round the clock

Mostly round the clock

Need for maintenance

Periodic per robot

Periodic for a platform that hosts a number of robots

Programming

Manufacturer specific

System / application specific

Response time

Depends on motor speed

Depends on server speed

Operating cost factors

Equipment cost, installation cost, maintenance charges (parts+labor)

Development cost, licensing cost, support fees (sometimes inclusive)

Integrate/communicate with other systems

Through hardware, adapters or interfaces

Through user interface

Lifecycle

Replaced with newer versions at end of life

Upgraded with newer versions at end of life

Ability to scan or read information

Through integrated equipment (e.g., NFC device / barcode reader)

Through connected external equipment (e.g., scanner)

IoT readiness

Partial (only newer equipment)

Full (all versions)

Example

Fanuc Palletizer

Infosys EdgeVerve AssistEdge

But these are not just the only two groups in which robots can be clubbed into. A third type has recently rolled out of the robot factory, well figuratively. I like to call it the hybrid variant - take a physical robot, load it up with some soft robot programming and add top if off with artificial intelligence. Although I tried to explain the concept like a simple drink recipe, am sure building an actual hybrid robot takes much more, rather substantially more, than this. Examples of these variants include, but not limited to:

  • Hanson Robotics' Sophia - the humanoid that was recently doing the rounds on internet. Interestingly, Sophia was also offered a citizenship by Saudi Arabia
  • IBM's Pepper - powered by their flagship AI platform, Watson. Over the past few months, Pepper is finding a place on the center stage at quite a few major symposiums, the most recent one being Oracle Modern Supply Chain Experience '18, where I was able to see it in action

Robotics has evolved in a big way since the time of its inception. But this decade has particularly seen a boom and the trend is expected to continue through the next few years. At the World Economic Forum in Davos, Switzerland this year (2018), Jack Ma, founder of Alibaba (China's largest e-commerce company) predicted that robots, coupled with AI and machine learning, are going to be able to handle an unbelievable number of tasks over the next decade or so. This might not have sounded logical maybe 5 to 6 years ago, but seems totally plausible now.

 

The Smart Ones

Even though robots have been around for a while now, the ability to have an actual conversation with these machines has taken a new dimension these days. From an 'I speak, the robot listens' model, this has transformed into 'I speak, the robot listens, the robot responds' and even further. You can now have meaningful nested conversations with a robot, where it weaves a series of questions and responses into a logical communication thread. Virtual assistants such as Apple's Siri or Amazon's Alexa or Microsoft's Cortana, Interactive Voice Response (IVR) based customer support agents, are some good examples. What's even more interesting is the self-learning potential that these smart robots can showcase with just some degree of artificial intelligence baked into them. I was amazed at how my Google Home Mini could perfectly pronounce my name (which is of Indian origin) with the correct accent from the moment it came to life. I never taught this, so how did it know? Another trait that intrigues me is when I ask for a song, it automatically plays it through my Spotify account on its built-in speaker. But when a movie trailer is asked, it sends a signal to Chromecast and plays it from the YouTube app on a connected TV! When in dilemma, its seeks clarification on my device preference, switching to it in a jiffy. This might sound like not a big deal for humans, but think from a little robot's perspective, which is just a tad larger in size than an ice hockey puck! These are just a couple of instances in everyday life which makes one wonder about the sheer intelligence of these machines.

Let's talk about chat bots, another form of smart soft robots. It is not uncommon that we visit a website and a little "helper" pops up asking "May I help you?" Behind the scenes, this robot has a technology that is able to assist us with basic queries without having to go through an actual agent. For example, checking the status of an order or shipment, troubleshooting tips for common issues with a device, providing contact information of the correct department, etc. What it is doing is simulating a person reading or hearing key words and searching a database for relevant answers. Of course, when things start to get a little over basic and cannot be handled by the bot, it will promptly connect us to a customer service representative. But at least this has eliminated the first layer of inquiry, and the best part is, generally there is no wait time to talk to a bot and you are always number 1 in sequence! With bots getting smarter, the line in sand, beyond which a bot has to pass on the baton, is going farther and farther.

In fact, any platform having a front-end user interface could be considered incomplete without an assistant bot. Maybe that is why in its latest release, Microsoft introduced a new bot feature called "Who" in their Office 365 Teams application. By asking it simple questions, the bot lets you search for anyone in your organization based on what they are working on, who they work with and more. Through a quick instant message (IM) conversation, you can get information about people in your organization such as:

- locate experts on a topic

- look up a person, their supervisor or their team members

- find people you had a meeting with about a particular topic, etc.

In essence, "Who" saves you the effort of going through a number of repositories such as the corporate directory, mailbox, calendars, etc. by doing that task on your behalf. What's more important is that it performs this search in multiple data sources simultaneously and presents the results in merely a second or two, quite faster than it would take us to look up the same in a traditional manner.

Just like chat bots are an usual thing on websites, another feature which is quite common is Completely Automated Public Turing Test, more commonly known to us as CAPTCHA. Although this is also entirely related to robots, the intent is exactly opposite of the robots we have discussed so far. The primary purpose of this is to prevent automated scripts (bots, smart but in a wrong way) from clogging the system by creating too many transactions or even potentially hacking it. Here are the two types of CAPTCHA that are most common:

  1. A check box that needs to be checked to confirm that the person accessing that site is not a robot, or in other words, is a human
  2. A mix of garbled or hazy characters / numbers or an image showing the same which one must enter accurately (at times, even in the right case) in an empty box

Now as robots are getting smarter, CAPTCHA needs to equally reciprocate. A noteworthy example of this is the website of a leading financial corporation in India. In addition to having users check a box or replicate a text, it presents them with mathematical or logical questions. Examples:

  • What is the sum of: 10 + 4 + 2
  • Which set of alphabets is in center of: Ma Ni Sh

These questions are quite basic, maybe comparable to elementary school syllabus, but the bottom line is only a human can read, process and answer them. Bad robots, stay out!


Automation with Robots

Robots have catapulted automation to levels that until now were seen only in sci-fi shows and movies. One of my neighbors, who installs heating and air-conditioning equipment for industries told me about his experience at a leading packaged drinking water manufacturing company here in Southeast Wisconsin. It was in the middle of a regular workday and he couldn't hear any human voices on the shop floor. Reason - The entire plant was being operated by robots! They traversed on electro-magnetic guiderails avoiding any collision and meticulously performed almost every task from bottling and packing to picking and loading orders in trucks. In some cases, they acted as per instructions relayed wirelessly by a few folks, while in some scenarios, the robots themselves were the decision makers. I was totally enthralled to hear this story and wanted to see something like this in action. To my luck, a couple of days later, I came across a video on LinkedIn showing how Alibaba's largest warehouse is fully operated by just 70 robots! These compact Quicktron robots not only lift and move pallets much larger than their size, but also aptly move to rest areas at the right time (read: dock themselves at charging points when running low on battery).

Couple of months ago, Dubai airport in collaboration with Emirates airlines did a pilot of a completely automated security screening system which they like to call it "the Aquarium". Passengers have to pass through a tunnel with augmented reality walls featuring bots in the shape of fish. While the passengers look around to experience the life-like aquarium, high definition cameras mounted on these fish bots take pictures (along with retina scans) and the tunnel itself x-rays the person. Biometric and scan data is then processed in real time to determine the outcome of this screening. The process is completed in a matter of seconds and security cleared passengers are on their way to board the flight while the flagged ones have to proceed to secondary checks. The test was a huge success and the system is expected to be fully functional soon. Another technological marvel made possible because of robotics and automation!

Soft robots have opened up newer avenues for automation. Robotic Process Automation, also known as RPA, is becoming increasingly popular with enterprises. A basic RPA framework comprises of a group of soft robots that can perform repetitive, non-judgmental tasks. Varying degrees of intelligence, analysis and decision-making abilities can also be built-in. Further, the solution can be extended by integrating peripheral components that can supply data from non-electronic sources. For example, if a company needs to key in large volume of paper invoices on a regular basis, then an OCR (optical character recognition) software such as AnyDoc can be bundled with a compatible document scanner. Subsequently, bots in a RPA solution (such as AssistEdge from Infosys EdgeVerve) would consume (read) these scanned invoices and enter them into the system (say, Oracle ERP) making the process completely automated, well almost. The only manual task would be someone having to place the set of paper invoices on the scanner's feed tray and replenishing it with a new batch once fully processed. Such RPA platforms can perform a multitude of tasks and are compatible with several systems. Usage of bots can be optimized by sharing them across business units, departments or job functions. Moreover, a bot is available round the clock and can be programmed to work on tasks of completely disparate nature by scheduling its runs. Multiple bots can also be deployed to work in parallel or in tandem, to get the biggest bang for your buck!

As capabilities of robots scale new heights dovetailed with investments in developing increasingly secure and smarter AI and machine learning platforms, our level of trust in these machines continues to grow proportionately. Google's driverless car project 'Waymo' is a real life, and perhaps the biggest, symbol of this. If you think about it, it is a robot in the shape of a car. Even though it looks way different than conventional robots (the ones with arms and all), it has all the functional characteristics of one. While the fully autonomous self-driving car is still in pilot stages (with the first few vehicles to undergo field / road tests in California in April 2018), several auto manufacturers are trying to bring in at least some flavor of robotics in their cars. More than adding to the enriched driving experience or comfort, the idea here is to make these cars safer. Features such as dynamic radar cruise control, lane departure correction, forward collision avoidance, auto-parking assist, etc. are being made available in every day passenger cars by Toyota, Nissan, Chevy and others through their Advanced Technology Package add-on. This is made possible by virtue of a powerful robotics system with sensors, radars, cameras and a software with ultra-fast real-time data processing capabilities integrated with the car's powertrain.

Let me explain with dynamic radar cruise control as an example. While this feature is engaged, if the vehicle in front slows down, radar sensors send a signal that physically depresses the brake pedal. This reduces your car's speed to maintain a safe distance from the deaccelerating vehicle in front of it. If this vehicle increases speed again or gets out of the way, the same set of sensors send another signal, but this time to the gas pedal thereby automatically accelerating your car to match the speed that was initially set on cruise control. The system basically replicates what we (as human drivers) would do in a similar situation and makes driving less stressful, and safer. Well, this robot always has its eyes on the road!

Inventors are getting creative with robotic automation trying to introduce it in our daily chores so that we can reap the benefits every single moment. Floor cleaning robots, such as iRobot, vacuum and mop meticulously covering each area smartly avoiding smashing into walls, furniture, pets and of course us! Companies like Samsung have even come up with sophisticated versions of these appliances that can be controlled with an app on your phone. Earlier this year, Chongqing supermarket in China introduced robotic shopping carts. The cart syncs with a smart phone and follows the shopper around the market. It not only saves them the hassle of pushing things around but also helps them locate stuff! This robotic cart even detects other shoppers and carts en-route and politely finds its way around them. Walmart is currently evaluating a prototype for a similar robotic cart. The point is, opportunities for automation are endless with the help of robotics. And engineers are leaving no stone unturned!


The Road Ahead

It is almost impossible to ignore robots these days. They are omnipresent in a plethora of shapes and forms. The truth is, we co-exist with robots, at times not realizing how important they are. Today, robotics cannot be seen in isolation. The world of robots, or "our world of robots" so to speak, is being made more and more exciting by other technologies that make them look good. As Elon Musk (Founder and CEO, Tesla Inc and SpaceX) once quoted, "When somebody has a breakthrough innovation, it is rarely one little thing. It's usually a whole bunch of things that collectively amount to a huge innovation." As quite a few technology leaders prudently envision, this is one area of innovation that is in its purple patch and it truly deserves to be there! This is going to stay and grow helping businesses achieve more in many years to follow.

In this article, I tried to gather my outlook and observations on this topic and collectively present them. This might not come across as a totally technical manuscript about implementing robotics, and that's intentional. The aim here was to jot down experiences that prove how robots have become an integral part of our lives, and that of a larger spectrum we deal with, but go unnoticed on many occasions because we fail to comprehend their importance. If you are equally passionate about this subject and would like to have a discussion, then send me an email at Manish_Naik@infosys.com and I would be happy to chat. Also, please share your feedback / questions in the comments section below or via email.

 

"See things in the present, even if they are in the future."

- Larry Ellison (Co-founder, Executive Chairman and CTO, Oracle Corporation)

Machine Learning - Introduction

0
0

Machine Learning - Introduction

Machine learning is a sub branch of Artificial Intelligence (AI) where we can write a code to use existing pre-defined algorithms to learn underlying patterns or underlying information in Raw data. Machine Learning basically means giving intelligence to machine to perform tasks as humans do. For example, now a days we hear a lot about google cars where we set destination and relax in car. The car will reach the destination without any other human intervention inspite of traffic / obstacles. The car is trained to identify obstacles/ traffic signals and act accordingly. Other examples applications include spam filtering, Credit card fraud detection, optical character recognition (OCR), Detecting faces in images, search engines, Handwriting recognition, Search engines, Handwriting recognition.... etc

  Types of Learning:

Machine learning tasks are divided or classified into mainly two categories, depending on the nature of the learning, "signal" or "feedback" available to a learning system.

These are:

Supervised learning

Unsupervised learning

Supervised learning:

Supervised learning is simply a formalization of the idea of learning from ex-supervised/ Pre-supervised examples. The computer is presented with example inputs and their desired outputs, given by a "teacher", and the goal is to learn a general rule that maps inputs to outputs. Here we observe the features X1, X2, . . . , Xp and we need to predict an associated response variable Y . Data is usually split into training, test and validation sets. Model gets trained using the trained set and predictions are then made on the test set and then validated with the actual output. Validation sets are then used to ensure models are not over-fitted.

The two sets of data are known as training set and a test set.  The idea is for the learner to "learn" from a set of labeled examples in the training set so that it can identify unlabeled examples in the test set with the highest possible accuracy.

For example, a training set might consist of images of different types of gadgets (say, laptop, mobile e.t.c.), where the identity of each gadgets in image is given to the learner. The test set would then consist of more unidentified pieces of gadget, but from the same classes. The goal for the learner to develop a rule that can identify the elements in the test set.


11g.JPG


Supervised methods can be used in many domains such as finance, manufacturing and marketing.

Supervised learning is further divided into two types.

1.) Regression

2.) Classification

 

            In Regression problems, we map the given data to real value domain. For Example, a regressor can predict the price of a house given its characteristics ( location, area, accessibility to roads, supermarkets, bus stops etc., ) . Some of the basic algorithms used in regression problems are Linear Regressor, Ridge Regressor, Lasso Regressor, Elastic Net, K-Nearest Neighbor Regressor, Decision Trees, Support Vector Machines, Artificial Neural Networks etc.

Claasifiers map the given data into pre-defined classed. For Exapmle, classifiers cab be used to classify whether loan can be given to a customer or not based on his credit score or whether an recieved email is genuine or spam etc., Some of the basic algorithms used are Logistic Regression, Linear Discriminant Analysis, Navie-Bayies classifier, K-Nearest Neighbour Classifier, Decision Trees, Support Vector Machines  e.t.c.,

22g.JPG

Un-Supervised learning:

                Unsupervised learning involves no target values. It tries to auto associate information from the inputs with an intrinsic reduction of data dimensionality. Here we observe only the features X1, X2, . . . , Xp and we are not interested in prediction, because we do not have an associated response variable Y. Unsupervised learning is based on the relations among the data, and is used to find the significant patterns or features in the input data without the help of a teacher.

For example, a dataset might consist of images of different types of gadgets (say, laptop, mobile e.t.c.), where the identity of each gadgets in image is not given to the learner. We need to design an algorithm so that the computer classifies the gadgets based on the size, shape or weight etc...

A criterion is needed to terminate the learning process. Without a stopping criterion, a learning process continues even when a pattern, which does not belong to the training patterns set, is presented to the network. The network is adapted according to a constantly changing environment. Hebbian learning, competitive learning, and SOM are the three well-known unsupervised learning approaches. Generally speaking, unsupervised learning is slow to settle into stable conditions.

              Un-Supervised learning is further divided into two types.

1.) Clustering

2.) Association

Clustering means grouping the data based on similarities between them. For Example, Let us assume that we had a library and we were given a lot of books as a donation. Now how can we arrange them in racks? We need to group certain books into categories and then arrange them in racks. Similar way clustering algorithms work. Clustering algorithms are used to take the data, find similarities and dissimilarities between them and group them together. 

44g.JPG

Clustering is normally done on distance based. Data is converted to set of points in space, with distance measures we group the points into number of clusters, such as within cluster or inter cluster distances should be minimum and intra cluster distances are maximum.

Association rule mining was used to discover interesting patterns. The items are stored in the form of transactions that can be generated by an external process, or extracted from relational databases or data warehouses. Market-basket analysis, one of the most intuitive applications of association rules, strives to analyze customer buying patterns by finding associations between items that customers put into their baskets. For example, you could mine the transaction data of a grocery store for frequent patterns and association rules. An example rule could be {milk, bread} -> {eggs}, this rule would tell you that if someone purchased milk and bread together, then they are also likely to purchase eggs.  Such information can be used as the basis for decisions about marketing activities such as, e.g., promotional pricing or product placements.

Zero Based Budgeting - A Rigorous way of Cost Optimization

0
0
 

Budgeting exercise in a typical organization starts with incrementing numbers from previous budgets, such as a 5% increase in expenses, without any rationalization for existing spending. A typical Budgeting manager would most of the times be increasing the expenses taking into consideration inflation.

Thus comes into picture the Zero Based Budgeting (ZBB) to make a reassessment of existing expenses and planning for new expenses from scratch. Traditional budgeting analyzes only new spending, while ZBB starts from zero and takes into consideration revised old inveterate expenses in addition to new expenditures. The main aim of ZBB is to put the responsibility on managers to provide explanation for expenses and it aims at driving value for the company by rationalizing costs and not just concentrating on revenue.

ZBB is essentially a cost control techniques, however it differs with the typical cost cutting approaches as usually they are a onetime effort while ZBB is a continuous ongoing process of rigorously resetting the cost bases. Taken as a whole, ZBB is a challenging and time-consuming activity which may give exceptional results when implemented rightly, whereas it may also have adverse impacts on employee morale due to its aggressive and demanding way of implementation. There may be reduction in Headcounts or some functions being eliminated, combined or outsourced. Thus it becomes particularly significant for the organization to find the correct balance to operate in a successful, sustainable way. By Implementing ZBB organizations have been able to increase visibility and identify cost reduction opportunities like headcount, SG&A, logistics, procurement, overhead expenses & IT etc.

Traditional Budgeting vs. ZBB

ZBB is a continuous ongoing process and it may not be required for every organization to implement ZBB. Decision to implement ZBB must be made after weighing the advantages and disadvantages:

Advantages

Disadvantages

Checking the Budget Inflation:  ZBB overcomes the limitation of incremental budgeting of inflation by rigorously checking each line item every year.

Desired Expertise: Relooking at processes each year would demand requisite expertise to understand each line item at the bottom level.

Rationalizing redundant Activities: The whole process of ZBB revolves around relooking at existing processes and rationalizing activities which have become redundant over time.

More Manpower for Budgeting exercise: Making the entire budget on ZBB may require the organization to increase the manpower to effectively bring the desired results.

 Accurate Numbers: As compared to traditional budgeting which involves changing numbers to previous budgets, ZBB demands each department to reassess their costs and thus are more accurate.

Increased time for Budgeting: Incremental budgets are easier methods which require less time as they don't require to go behind each line item.

Efficient use of resources:  Rather than looking at existing headcount numbers, ZBB calls for appointing individuals with clear accountability for each activity.

Delayed response Time: time and training required to do ZBB is higher which may result in  staff getting late in responding to a changing market. 

Efficient: It helps by focusing on current numbers instead of past budgets.

 

 

Steps for Zero Based Budgeting:

 

Building Apex applications on Oracle Database

0
0

Oracle Application Express ( Oracle APEX )

What is Oracle Application Express

Oracle Application Express widely  referred as Oracle APEX also, is a  feature provided by Oracle on its Database which enables user build web based applications using SQL and PL/SQL.  Any developer with SQL and PL/SQL skills can  design, built and deploy database-driven applications using its browser based user interface. APEX features such as pre-built user interface themes (templates), navigational controls and form handlers among others makes application development fast and easy  and also  eliminates dependency  on UI developers/experts.

Apex Architecture

Apex is installed /configured  on an Oracle database instance and consists of data in the schema and PL/SQL code written for the business logic of application.

 

Apex application and page are accessed from browser. These requests are then channeled through the middle-tier for execution in the Oracle Database and returned as HTML responses to the browser.

Mid tier plays no role in data manipulation or processing, it is the  Oracle APEX engine which ( a part of Oracle Database) accepts  page, and interacts with the schemas in the database.

The mid-tier can be

1. Oracle REST Data Services (ORDS) running on a Java Server ( Oracle Weblogic Server, Oracle Glassfish, Tomcat )

2. Oracle embedded PL/SQL Gateway (EPG).

3.  Oracle HTTP Server ( with mod_plsql ).

 

Advantage Apex

It is a good alternative to Oracle forms.

Leverages oracle database capabilities and sql and PL/SQL.

Capability to load data in database tables from .csv and .xlx files.

No separate license required, it comes free with Oracle DB.

Easy to develop application on apex and deploy across environments.

 

Apex Integration with OBIEE

 Lets first understand apex URL syntax and what each parameter in the syntax does.

Syntax of an APEX page URL follows the format as below:

http://host_name/pls/apex/f?p=AppId:PageId:Session:Request:Debug:ClearCache:Params:ParamValues:PrinterFriendly

 

http://  - It can  be either of  http or https protocol.

host_name  -  domain/host/server, it can also be localhost. IP address and port number can  be part of it as well.

/pls - This indicates use of Oracle HTTP Server.

/apex - This is refers to name of Database Access Descriptor (DAD). DAD describes how the HTTP server connects to DB server to service  HTTP request ( default value being apex ).

/f?p -  f here public  procedure and main entry point to APEX, and  'p' is parameter to it. (the entire string which consists of below is parameter to 'f').

:AppId - This refer to  application Id or alphanumeric alias of apex application being accessed.

:PageId - This is  page number or alphanumeric alias of page which is called in the URL ( Every page is part of an apex application ).

:Session - This is apex  unique Session ID. It can be empty (then APEX creates a new Session) or set to 0 for Public Pages.

:Request - This is a keyword and used to set value of REQUEST and control page behavior. On click  application button sets the value of REQUEST to the name of the button.

:Debug --  This is used to enable or disable debug mode for the page. Valid values are YES or NO.  

:ClearCache --  This is used to clear cache and set values of page items to null.  Provide list of page ids for which cache is to be cleared.

:Parameters -  Multiple Parameters (page items separated by comma) can be passed from URL  to apex age for page items as well as application items

 :ParamValues -  This contains values (comma separated in case of multiple parameters) for the parameter list in the above. Values cannot have comma and colon as part of it.

:PrinterFriendly - To  render an apex  page  in printer friendly  mode, set value as 'Yes' for this parameter.

 

An apex application page can accessed  from  or integrated with OBIEE in multiple ways:


1.  Calling apex page from report column using interaction feature.


 

We can define parameter names and respective values. Value can be hardcoded, column value from the report or value of a variable.

 

2. Apex page embedded in OBIEE dashboard.  

 

How and where we used Apex

We  can built simple as well complex application using apex.

 We Automated  some of the manual ( dependent ) tasks and integrated to OBIEE.

  1. Apex base application to manage Business reference data : Using this apex page authorized user can manage (modify , create, edit ) , refer to business metadata in the database table and reducing the dependency on DBA/ backend team.

    .



2.Application to retarget orders due for fulfillment.

Apex application was created to retarget current and subsequent stages of an under progress order and capture the audit information of retarget.

 


3. Created an apex application page to provide user ability to add comments/inputs for delayed orders which can be seen by different stakeholders. This task was performed over emails using microsoft excel.



Hadoop Overview

0
0

 

This blog deals with the Architecture of Hadoop, advantages and disadvantages of Hadoop.

------------------------------------------------------------------------------------------------------------------------------------------

 

Let's first understand what is Hadoop?

Hadoop is open source framework for processing and storing large data sets across the different clusters of computer which are present in different geographical locations.

Now let's understand why Hadoop?

The problem with the traditional database management systems is that it can process only structured data and it can handle only small amount of data (giga bytes). Hadoop can handle structured, unstructured and semi structured data. Hadoop can handle large amounts of data with high processing speed through parallel processing.

The Architecture of Hadoop has mainly two components. They are

1.       Hadoop Distributed File System - For Storing Data

2.       Map Reduce - Processing

MRArchitecture.png

 

Name Node is the master node which does the tasks like memory management, process management. It is the single point of failure in Hadoop Cluster. Secondary Name Node takes the backup of the namespace of the Name Node and updates the edits file into the FSimage file periodically. Data Nodes are the slave nodes which does the computations.

When client submits the job to Name Node, it divides the files into chunks and distributes the chunks to Data Nodes for processing. Each chunk is replicated 3 times and will be stored on three different Data Nodes. If one Node is going down, then the Name Node identifies the Data Node which have the replicated file and starts execution. This process makes Hadoop a fault tolerant System.

Now let's discuss the Limitations of Hadoop

1.       Handling small files:

If you want to process large number of small files, then Name Node needs to store the HDFS location of each file. This will become over head for the Name Node. This is the reason why Hadoop is not recommended when it comes to handling large number of small files.

 

2.       Processing Speed:

To process large datasets MapReduce follows Map and Reduce mechanism. During this process, the intermediate results of Mapper, Reducer function are stored to HDFS Location which results in the increase of I/O operations. Thus, the processing speed get decreased.

 

3.       Not able to Handle Real Stream Data:

Hadoop can process large amount of batch files very efficiently. When it comes to Real Stream processing, Hadoop failed handle the real-time data.

 

4.       Not Easy to Code:

Developers need to write code for each operation they need to perform on data, which makes it very difficult for them to work.

 

5.       Security:

Hadoop does not provide proper authentication for accessing the cluster and it does not provide any information about who has accessed the cluster and what data the user has viewed. Security is the biggest draw back when it comes to Hadoop.

 

6.       Easy to Hack:

Since Hadoop is written in Java, which makes cyber criminals to hack the system very easily.

 

7.       Caching:

There is no cache mechanism in Hadoop for storing the intermediate results for further use. As result of this the performance got diminished.

 

8.       Line of Code:

The line of code for Hadoop 1,20,000, which makes it difficult for debugging and executing.

 

9.       Unpredictability:

In Hadoop we can't guarantee the time for completion of job.


Big Data Processing Architecture

0
0
 

 

In today's world, business needs to take decisions instantly on the data provided by business analyst to stay on top. In current scenario business analyst needs to process and analyze all types of data (structured, semi-structured and unstructured) in short span of time, which is not possible only through the traditional data warehouse concepts, to achieve this we need to move to big data. When we have decided to move to big data, which is the best architecture we can implement?

The two widely used big data processing architecture

  • Lambda architecture

  • Kappa architecture 

Lambda Architecture:

The lambda architecture ensures both batch and real time data are taken into consideration for analysis. Using this architecture history of the data is maintained for any future analysis. The lambda architecture has the following advantages:

  • latency time

  • throughput

  • volatile

Latency time:

Latency time is the time taken between the generation of data and its availability in the reporting layer for analytics.

Throughput:

The large volume of data is broken into small blocks, gets processed in parallel increasing the throughput.

Fault tolerance:

The system which is designed to continue processing even when there is some error. 

 

Architecture diagram:

The three major components of Lambda architecture:

  • Batch layer

  • Speed layer

  • Serving layer

 

Batch layer:

In the batch layer the complete data will be stored in HDFS immutable system, which means the data can only be appended not updated or deleted. The versioning of the data in append logic is achieved using the time stamp. From the main file system, the by using the Map reducer the data will be pre-commute to the batch views as per the business requirement. The ad-hoc querying is also possible on the batch view. The batch view is generated to achieve low latency time.

Apache Hadoop is used in the batch layer processing.

Speed layer:

The records which are missed in the latency time of batch processing will be fetched and stored in the speed layer dataset. Speed layer is a delete and create dataset, once the batch layer processing is completed speed layer dataset will be deleted and created with new data. From the speed layer dataset by using complex algorithm real time views are created.

Apache storm, SQL stream is used in speed layer processing. 

Serving layer:

The serving layer is where the user queries for output. Once the user query is fired batch view and real-time view outputs are combined and near real time output will be provided. Druid can be used in the serving layer to handle both batch and speed layer views. 

Kappa Architecture:

Kappa architecture is nothing but a simple Lambda architecture system with batch layer removed. This architecture has been designed in such a way that the speed layer is capable enough to handle both real time and batch data.

 Architecture diagram:

The Apache Kafka streaming data is specifically designed to achieve Kappa architecture. In Kappa architecture there is only one code which needs to maintained for stream processing (Apache Kafka). In kappa architecture we need to run the full load job at first time of development as well any code change. Once the reprocessing of the full data is done and presented in the serving layer, the speed layer job will run fetching the latest records and present it in the serving layer. The speed layer job can be designed using (Apache Storm, Apache Samza, and Spark Streaming). The serving layer can use no sql database, apache drill etc.

Which architecture to go for?

  • If the code logic for both the batch layer and real time layer are same then we can go for kappa architecture.

  • If the code logic of batch layer and speed layer are different then we can go for lambda architecture.

ETL vs. ELT

0
0

In today's world, the process of innovative data warehousing is going through a rapid transformation like big data, sensor data, social media data, endless behavioral data about website/mobile app users etc. This gives rise to various new methodologies, for example, ELT is heard progressively in today's analytical environment. We have used ETL in the past but what happens when the "T" and "L" are switched since they both solve the same need.

Avoid Validation failures with DRM Validation checks

0
0

Avoid Validation failures with DRM Validation checks

Data Relationship Management (DRM) as a master data management solution has been a favorite in a complex EPM and EBS landscape for its audit and versioning features along with other unique features. Now that DRM is available in cloud, its adaption for customers is expected to increase multifold. This blog would help you in understanding validation checks configuration in DRM that would help in finding any issues before your target systems get impacted.

Whenever a metadata update is done in DRM, it is advisable to do a few checks. The most vital of these checks is to ensure that all the target system exports that may be affected by the metadata update are frisked for any validation failures. This is especially important when new members are added or any existing members are deleted. This blog talks in detail about the process of validating the exports and hierarchies after a masterdata update. It also highlights how to go about fixing a validation failure, if there is any.

You may ask as to why is this this important? DRM Hierarchies and Exports are used for supplying various downstream systems with masterdata. These hierarchies and target system exports in DRM have validations applied to them. These validations are to ensure that certain business requirements are being met during masterdata maintenance. So, whenever there is a masterdata update done to the system, it becomes imperative for the DRM Admins to validate all the hierarchies and exports that may have got affected by the update.


Detailed Process for Validation Check:

Doing the validation check is a very simple process. Let us assume that some metadata update was done in the 'Total_AuditTrail' hierarchy. So, our next steps would be:

  • Open the related exports. In this case, let us assume that the export that is affected is 'HFM_AuditTrail_ParentChild'.

  • Click on the 'Run Export Validations' icon at the top left corner of the screen (shown below as well).

 

  • DRM would run a check to ensure that the metadata changes are meeting the criterion of all the validations that are applied on the Hierarchy or the export. If all the validations are being met, we will get a success message like below.


Note: Running the validation check does not mean that we are exporting anything to any target system, and is completely unrelated to any metadata being exported to the target. So, feel free to run this check as many times as required.

Now, let's consider a scenario in which the Validation check returns error.


  • Let us assume that we are running the validation check on the export HFM_Area_MemberAttributes'. Click the 'Run Export Validations' icon.

  • DRM would run a check to verify whether all the validations are being met and return you to the home screen.



 


 



  • Click on the 'Validation Results' tab.


  • The Validation Results tab will highlight all the members which have failed the validations check.

     

  • Click on the Go icon to be directed to the nodes that have failed the validation check.

 

  • Clicking on the GO Icon again will take you to the position of that particular member in the Hierarchy.

The above process will help you in identifying the error within minutes so that it may be fixed at the earliest. After making the required modifications, run the validation checks again to ensure that the export has no more validation failures.

Alternately, you may also run the Validation Check at the Hierarchy Level as well.

  • In the below example, let us assume that we had made some metadata updates in the ar.NVCLS hierarchy. When you have finished making the metadata changes, select the top member of the hierarchy, and right click. In the validate option, select 'Assigned'.


 


 


 


 


 



  • This will also highlight the validation errors in the same way.
  •  

    Note: Running the validation check whenever metadata is updated is a best practice suggested by Oracle.

Automate your File based Data Loads in FDMEE with Open Batches

0
0

Automate your File based Data Loads in FDMEE with Open Batches


In one of my recent implementation assignments, we were faced with the requirement of eliminating any manual intervention and automating our data loads in Financial Data Management Enterprise Edition (FDMEE), the target system being Hyperion Financial Management (HFM) and Hyperion Tax Provisioning (HTP). Open Batches in FDMEE is an effective way of automating your file based data loads. Though the standard User Guides contain precious little on creating and running Open Batches and are not entirely comprehensive, Open Batches are easy to use and implement. In this blog, I attempt to explain in detail the process of creating Open Batches. These details have come purely from my personal research and my experience from my past assignments.


 


What is an Open Batch?


Batches play a critical role in automating the Data/Metadata load process. By creating batches, we can easily club different rules together or execute different rules in an automated way by leveraging the scheduling capabilities of FDMEE.



An Open Batch is instrumental in importing data into a POV in FDMEE from a file based data source. Open Batches can be automated to run at scheduled timings. Most importantly, Open Batches pitch in when we have to load more than 1 data file for any particular location. Also, Open Batches provide the flexibility of loading these multiple data files in 'Serial' or 'Parallel' mode.



Configuring and Executing an Open Batch


Let us assume that we have 2 FDMEE Locations and from each of these locations, we have to load 2 data files:


Step 1: Setting up the required FDMEE Locations


We create 2 locations as shown below (without deep-diving into the details of an FDMEE Location)

a. LOC_Sample_Basic1

b. LOC_Sample_Basic2




Step 2: Setting up the Open Batches



A new Open Batch may be created from: Setup -> Batch -> Batch Definition. Click on 'Add' to add a new Batch Definition.



Key in the basic details such as NAME (Name of the Open Batch), TARGET APPLICATION, WAIT FOR COMPLETION and DESCRIPTION. Apart from these, there are a few other details as below:


 


  1. Type: Select 'Open Batch'.

     

  2. Execution Mode: Select 'Serial' if you wish to load all the data files in that location one by one. Select 'Parallel' for loading all the files simultaneously.

     

  3. Open Batch Directory: This is where the User Guides are specifically vague. All we need to do is to specify the name of the folder where the data files will be placed. But, we need to take special care that this folder is created at a level which is lower than the level of the folder 'openbatch'. So, assuming that we need to place the data files in a folder 'Test1', this folder will be created as shown below:


  1. File Name Separator: For File Name Separator, we have to choose from (~ , _ , @ or : ).

     

  2. Naming convention of the Data Files: For a data file to be executed from an Open Batch, it needs to have the following components in it name, separated by the chosen 'File Name Separator':

     


  • File ID: free-form value used to sort the files for a given Location in an order. The values for this can be 1,2,3... or a,b,c... etc, just to specify the order in which the files will be loaded.

  • Data Load Rule Name: The name of the Rule through which the file will be loaded

  • Period: The period for which the file is to be loaded

  • Load Method: 2-character value where first character denotes 'Import Mode' (possible values 'A' and 'R' for 'append' and 'replace' respectively). The second character denotes 'Export Mode' (possible values 'A', 'R', 'M' and 'S' for 'accumulate', 'replace', 'merge' and 'replace by security' respectively).


So, now let us assume that we have to load 2 data files from the location 'LOC_Sample_Basic' using the Data Load Rule 'DLR_Sample_Basic'. Assuming that the 'File Name Separator' chosen is '@', the file names for the 2 files will be:


  • 1@DLR_Sample_Basic@Jan-17@RR

  • 2@DLR_Sample_Basic@Jan-17@RR


  1. The 2 Open Batches for loading the data files will be as below:

OP_Sample_Basic1

OP_Sample_Basic2




 


Step 3: Configuring the Master Batch


After doing all the above configurations, the default question would be that 'Where is the automation in this when we have ended up creating 2 or more Open Batches? How will we execute 2 or more Open Batches without Manual Intervention?'. The answer to this query lies with the creation of the open batch type 'Batch'. We configure an open batch with the type as 'Batch' and name it 'BatchMaster'.



The feature of the Batch is that we can add multiple Open Batches to the Batch Master and then, schedule just this one batch instead of worrying about multiple Open Batches. To add the 2 Open Batches that we had created earlier to this BatchMaster, click the 'Add' button and enter the details. Ensure that you enter the Job Sequence number to tell the system which Open Batch to execute first.



 


Step 4: Executing the Open Batches


The Open batches or the BatchMaster can be executed from: Workflow -> Other -> Batch Execution


You may simply select the BatchMater now and schedule it easily through the 'Schedule' button.



FDMEE Multi-Period Data Load


There may be a requirement to load data files that contain data/amount values for multiple periods. Files for an open batch multiple period load are stored in 'inbox\batches\openbatchml' directory. There are just a few minor variations between setting up a Batch for Multi-Period Load and a Single Period Load.


 


  1. Open Batch File Name: The file name for multi-period files should be as per the format 'FileID_RuleName_StartPeriod_EndPeriod_Loadmethod':


    • FileID: free-form value used to define the load order

    • RuleName: Name of the data load rule which will be used to process the file

    • StartPeriod: First period in the data file

    • EndPeriod: Last period in the data file

    • Load Method: Import and Export Mode

      So, a multi-period file name would look like for a Data Load Rule named 'Actuaload':


    • 1_Actuaload_Jan-17_Dec-17_RR

    • 2_Actuaload_Jan-17_Dec-17_AR

       

       


  1. Open Batch Directory: We specify the name of the folder in the same way as we did for single period batch. But, we need to take special care that this folder is created at a level which is lower than the level of the folder 'openbatchml' instead of 'openbatch'. So, assuming that we need to place the data files in a folder 'Test3', this folder will be created as shown below:



 


  1. Sample Open Batch for Multi-Period:


Viewing all 561 articles
Browse latest View live




Latest Images