Things to know about web scraping

Things to know about web scraping

First things first, it is important to understand what web scraping means and what is its purpose. Web scraping is a computer software technique through which people can extract information and content from various websites. The main purpose is to use that information in a way that the site owner does not have direct control over it. Most people use web scraping in order to turn commercial advantage of their competitors into their own.

There are many scraping tools available on the Internet, but because some people might think that web scraping goes long beyond their duties, many small companies that provide this type of services have appeared on the market. This way, you can turn this challenging and complex process into an easy web scraping one, which, believe it or not, exists for nearly as long as the web. All you have to do is some quick research on the Internet and find the best consultant that is willing to help you with this matter. When it comes to the industries that web scraping is targeting, it is worth mentioning that some of them prevail over others. One good example is digital publishers and directories. They are one of the easiest targets for web scrappers, because most of their intellectual property is available to a large number of people. Industries like travel or real estate are also a good place for scraping, along with ecommerce, which is an obvious target too. Time-limited promotions and even flash sales are the reasons why ecommerce is seen as a candy by web scrapers.

Source: http://www.amazines.com/article_detail.cfm/6196289?articleid=6196289

Data Mining Basics

Data Mining Basics

Definition and Purpose of Data Mining:

Data mining is a relatively new term that refers to the process by which predictive patterns are extracted from information.

Data is often stored in large, relational databases and the amount of information stored can be substantial. But what does this data mean? How can a company or organization figure out patterns that are critical to its performance and then take action based on these patterns? To manually wade through the information stored in a large database and then figure out what is important to your organization can be next to impossible.

This is where data mining techniques come to the rescue! Data mining software analyzes huge quantities of data and then determines predictive patterns by examining relationships.

Data Mining Techniques:

There are numerous data mining (DM) techniques and the type of data being examined strongly influences the type of data mining technique used.

Note that the nature of data mining is constantly evolving and new DM techniques are being implemented all the time.

Generally speaking, there are several main techniques used by data mining software: clustering, classification, regression and association methods.

Clustering:

Clustering refers to the formation of data clusters that are grouped together by some sort of relationship that identifies that data as being similar. An example of this would be sales data that is clustered into specific markets.

Classification:

Data is grouped together by applying known structure to the data warehouse being examined. This method is great for categorical information and uses one or more algorithms such as decision tree learning, neural networks and "nearest neighbor" methods.

Regression:

Regression utilizes mathematical formulas and is superb for numerical information. It basically looks at the numerical data and then attempts to apply a formula that fits that data.

New data can then be plugged into the formula, which results in predictive analysis.

Association:

Often referred to as "association rule learning," this method is popular and entails the discovery of interesting relationships between variables in the data warehouse (where the data is stored for analysis). Once an association "rule" has been established, predictions can then be made and acted upon. An example of this is shopping: if people buy a particular item then there may be a high chance that they also buy another specific item (the store manager could then make sure these items are located near each other).

Data Mining and the Business Intelligence Stack:

Business intelligence refers to the gathering, storing and analyzing of data for the purpose of making intelligent business decisions. Business intelligence is commonly divided into several layers, all of which constitute the business intelligence "stack."

The BI (business intelligence) stack consists of: a data layer, analytics layer and presentation layer.

The analytics layer is responsible for data analysis and it is this layer where data mining occurs within the stack. Other elements that are part of the analytics layer are predictive analysis and KPI (key performance indicator) formation.

Data mining is a critical part of business intelligence, providing key relationships between groups of data that is then displayed to end users via data visualization (part of the BI stack's presentation layer). Individuals can then quickly view these relationships in a graphical manner and take some sort of action based on the data being displayed.

Source:http://ezinearticles.com/?Data-Mining-Basics&id=5120773

Data Mining and Financial Data Analysis

Introduction:

Most marketers understand the value of collecting financial data, but also realize the challenges of leveraging this knowledge to create intelligent, proactive pathways back to the customer. Data mining - technologies and techniques for recognizing and tracking patterns within data - helps businesses sift through layers of seemingly unrelated data for meaningful relationships, where they can anticipate, rather than simply react to, customer needs as well as financial need. In this accessible introduction, we provides a business and technological overview of data mining and outlines how, along with sound business processes and complementary technologies, data mining can reinforce and redefine for financial analysis.

Objective:

1. The main objective of mining techniques is to discuss how customized data mining tools should be developed for financial data analysis.

2. Usage pattern, in terms of the purpose can be categories as per the need for financial analysis.

3. Develop a tool for financial analysis through data mining techniques.

Data mining:

Data mining is the procedure for extracting or mining knowledge for the large quantity of data or we can say data mining is "knowledge mining for data" or also we can say Knowledge Discovery in Database (KDD). Means data mining is : data collection , database creation, data management, data analysis and understanding.

There are some steps in the process of knowledge discovery in database, such as

1. Data cleaning. (To remove nose and inconsistent data)

2. Data integration. (Where multiple data source may be combined.)

3. Data selection. (Where data relevant to the analysis task are retrieved from the database.)

4. Data transformation. (Where data are transformed or consolidated into forms appropriate for mining by performing summary or aggregation operations, for instance)

5. Data mining. (An essential process where intelligent methods are applied in order to extract data patterns.)

6. Pattern evaluation. (To identify the truly interesting patterns representing knowledge based on some interesting measures.)

7. Knowledge presentation.(Where visualization and knowledge representation techniques are used to present the mined knowledge to the user.)

Data Warehouse:

A data warehouse is a repository of information collected from multiple sources, stored under a unified schema and which usually resides at a single site.

Text:

Most of the banks and financial institutions offer a wide verity of banking services such as checking, savings, business and individual customer transactions, credit and investment services like mutual funds etc. Some also offer insurance services and stock investment services.

There are different types of analysis available, but in this case we want to give one analysis known as "Evolution Analysis".

Data evolution analysis is used for the object whose behavior changes over time. Although this may include characterization, discrimination, association, classification, or clustering of time related data, means we can say this evolution analysis is done through the time series data analysis, sequence or periodicity pattern matching and similarity based data analysis.

Data collect from banking and financial sectors are often relatively complete, reliable and high quality, which gives the facility for analysis and data mining. Here we discuss few cases such as,

Eg, 1. Suppose we have stock market data of the last few years available. And we would like to invest in shares of best companies. A data mining study of stock exchange data may identify stock evolution regularities for overall stocks and for the stocks of particular companies. Such regularities may help predict future trends in stock market prices, contributing our decision making regarding stock investments.

Eg, 2. One may like to view the debt and revenue change by month, by region and by other factors along with minimum, maximum, total, average, and other statistical information. Data ware houses, give the facility for comparative analysis and outlier analysis all are play important roles in financial data analysis and mining.

Eg, 3. Loan payment prediction and customer credit analysis are critical to the business of the bank. There are many factors can strongly influence loan payment performance and customer credit rating. Data mining may help identify important factors and eliminate irrelevant one.

Factors related to the risk of loan payments like term of the loan, debt ratio, payment to income ratio, credit history and many more. The banks than decide whose profile shows relatively low risks according to the critical factor analysis.

We can perform the task faster and create a more sophisticated presentation with financial analysis software. These products condense complex data analyses into easy-to-understand graphic presentations. And there's a bonus: Such software can vault our practice to a more advanced business consulting level and help we attract new clients.

To help us find a program that best fits our needs-and our budget-we examined some of the leading packages that represent, by vendors' estimates, more than 90% of the market. Although all the packages are marketed as financial analysis software, they don't all perform every function needed for full-spectrum analyses. It should allow us to provide a unique service to clients.

The Products:

ACCPAC CFO (Comprehensive Financial Optimizer) is designed for small and medium-size enterprises and can help make business-planning decisions by modeling the impact of various options. This is accomplished by demonstrating the what-if outcomes of small changes. A roll forward feature prepares budgets or forecast reports in minutes. The program also generates a financial scorecard of key financial information and indicators.

Customized Financial Analysis by BizBench provides financial benchmarking to determine how a company compares to others in its industry by using the Risk Management Association (RMA) database. It also highlights key ratios that need improvement and year-to-year trend analysis. A unique function, Back Calculation, calculates the profit targets or the appropriate asset base to support existing sales and profitability. Its DuPont Model Analysis demonstrates how each ratio affects return on equity.

Financial Analysis CS reviews and compares a client's financial position with business peers or industry standards. It also can compare multiple locations of a single business to determine which are most profitable. Users who subscribe to the RMA option can integrate with Financial Analysis CS, which then lets them provide aggregated financial indicators of peers or industry standards, showing clients how their businesses compare.

iLumen regularly collects a client's financial information to provide ongoing analysis. It also provides benchmarking information, comparing the client's financial performance with industry peers. The system is Web-based and can monitor a client's performance on a monthly, quarterly and annual basis. The network can upload a trial balance file directly from any accounting software program and provide charts, graphs and ratios that demonstrate a company's performance for the period. Analysis tools are viewed through customized dashboards.

PlanGuru by New Horizon Technologies can generate client-ready integrated balance sheets, income statements and cash-flow statements. The program includes tools for analyzing data, making projections, forecasting and budgeting. It also supports multiple resulting scenarios. The system can calculate up to 21 financial ratios as well as the breakeven point. PlanGuru uses a spreadsheet-style interface and wizards that guide users through data entry. It can import from Excel, QuickBooks, Peachtree and plain text files. It comes in professional and consultant editions. An add-on, called the Business Analyzer, calculates benchmarks.

ProfitCents by Sageworks is Web-based, so it requires no software or updates. It integrates with QuickBooks, CCH, Caseware, Creative Solutions and Best Software applications. It also provides a wide variety of businesses analyses for nonprofits and sole proprietorships. The company offers free consulting, training and customer support. It's also available in Spanish.

Source:http://ezinearticles.com/?Data-Mining-and-Financial-Data-Analysis&id=2752017