Showing posts with label scraping. Show all posts
Showing posts with label scraping. Show all posts

Monday, May 29, 2017

Know How to Consuming SOAP(asmx) Web Service through Android

This article will help you call a SOAP web service in the simplest way. We will pass a parameter to the web service an will receive it's result. In this tutorial I have used w3school's TempConvert web service. It's publicly available to everyone to consume the web service.

Before we begin with the code, we would to have download KSOAP library for our android project. The library used in this article is ksoap2-android-assembly-2.6.1-jar-with-dependencies.jar. To include this library in your project - Right Click on the project> Select Properties> Java Build Path> Libraries Tab> Add External JARs and then select your downloaded library. Now, on the Order and Export Tab checkmark the checkbox for this library.

The web service has two methods i.e CelsiusToFahrenheit and FahrenheitToCelsius so we have to design a simple android activity for the same.

Layout - main.xml

<?xml version="1.0" encoding="utf-8"?>
<LinearLayout xmlns:android="http://schemas.android.com/apk/res/android"
    android:layout_width="fill_parent"
    android:layout_height="fill_parent"
    android:orientation="vertical" >

    <TextView
        android:id="@+id/textView1"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="TempConvert"
        android:layout_gravity="center"
        android:layout_marginBottom="20dp"
        android:textAppearance="?android:attr/textAppearanceLarge" />

    <TextView
        android:id="@+id/textView2"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Input:" />

    <EditText
        android:id="@+id/editText1"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:inputType="text"
        android:layout_marginBottom="5dp"
        android:ems="10" >

        <requestFocus />
    </EditText>

    <TextView
        android:id="@+id/textView3"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Convert To:" />

    <RadioGroup android:id="@+id/rgTemp"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content">
    
    <RadioButton
        android:id="@+id/radioButton1"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Celsius" />

    <RadioButton
        android:id="@+id/radioButton2"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:layout_marginBottom="5dp"        
        android:text="Fahrenheit" />
    
    </RadioGroup>

    <TextView
        android:id="@+id/textView4"
        android:layout_width="wrap_content"
        android:layout_height="wrap_content"
        android:text="Result:" />

   <EditText
        android:id="@+id/editText2"
        android:layout_width="match_parent"
        android:layout_height="wrap_content"
        android:inputType="text"
        android:editable="false"
        android:ems="10" />

</LinearLayout>
 

To make a call to a SOAP web service, we mainly need four variables i.e namespace, method_name, address and soap_action. Here's how to get them.  


Now, we have to pass the parameter to web to get the result. We use the PropertyInfo class of the KSOAP library to pass the value and it's description.
PropertyInfo pi=new PropertyInfo();
pi.setName(PROPERTY_NAME);
pi.setValue(val);
pi.setType(String.class);
request.addProperty(pi);

We have set a setOnCheckedChangeListener on our radio group. So, whenever user selects an option we will take the value from the input edittext and then call our function which will pass the user input to the web service. Also, if user selects convert to Fahrenheit(or vice-versa) that means he has entered a Celsius value wants to convert it to Fahrenheit. Hence, we will call CelsiusToFahrenheit method of the web service and change variables accordingly.

Here's is the whole code-

package com.loginworks.demo;

import org.ksoap2.SoapEnvelope;
import org.ksoap2.serialization.PropertyInfo;
import org.ksoap2.serialization.SoapObject;
import org.ksoap2.serialization.SoapSerializationEnvelope;
import org.ksoap2.transport.HttpTransportSE;
import android.app.Activity;
import android.os.Bundle;
import android.widget.EditText;
import android.widget.RadioButton;
import android.widget.RadioGroup;

public class SOAPDemoActivity extends Activity {
    EditText input,result;
    RadioButton cel,fah;
    RadioGroup rGroup;
    String PROPERTY_NAME;
    
    public  String SOAP_ACTION;

    public  String METHOD_NAME; 

    public  final String WSDL_TARGET_NAMESPACE = "http://tempuri.org/";

    public  final String SOAP_ADDRESS = "http://www.w3schools.com/webservices/tempconvert.asmx";
    
    /** Called when the activity is first created. */
    @Override
    public void onCreate(Bundle savedInstanceState) {
        super.onCreate(savedInstanceState);
        setContentView(R.layout.main);
        
        input = (EditText) findViewById(R.id.editText1);
        result = (EditText) findViewById(R.id.editText2);
        
        cel= (RadioButton) findViewById(R.id.radioButton1);
        fah= (RadioButton) findViewById(R.id.radioButton2);
        
        rGroup = (RadioGroup) findViewById(R.id.rgTemp);
        

       rGroup.setOnCheckedChangeListener(new RadioGroup.OnCheckedChangeListener()  {
            public void onCheckedChanged(RadioGroup rGroup, int checkedId)  {
                
            String in = input.getText().toString();
            
               if (cel.isChecked()) {
                   PROPERTY_NAME = "Fahrenheit";
                   METHOD_NAME = "FahrenheitToCelsius";
                   SOAP_ACTION  = "http://tempuri.org/FahrenheitToCelsius";                   
               }
               else {
                   PROPERTY_NAME = "Celsius";
                   METHOD_NAME = "CelsiusToFahrenheit";
                   SOAP_ACTION  = "http://tempuri.org/CelsiusToFahrenheit";
               }
                                           
               Convert(in);
               
            }            
        });        
      }   
    
    public void Convert(String val) {
         SoapObject request = new SoapObject(WSDL_TARGET_NAMESPACE, METHOD_NAME);
         
         PropertyInfo pi=new PropertyInfo();

             pi.setName(PROPERTY_NAME);
                pi.setValue(val);
                pi.setType(String.class);
                request.addProperty(pi);
                         
             SoapSerializationEnvelope envelope = new SoapSerializationEnvelope(SoapEnvelope.VER11);
             envelope.dotNet = true;

             envelope.setOutputSoapObject(request);

             HttpTransportSE httpTransport = new HttpTransportSE(SOAP_ADDRESS);

             Object response= null;
             
             try    {
                 
             httpTransport.call(SOAP_ACTION, envelope);
             response = envelope.getResponse();

                 }
             catch (Exception exception) {   
                 
                 response=exception;                     
                 }
                 
             result.setText(response.toString());
    }
}



Note: When you will select an option, you would notice that your activity gets stuck for few seconds that is because we are working with network on the UI thread. So, it's advisable to create a new thread or AsyncTask for network related functions.
Also, don't forget to register your Activity in your Manifest file and add the following permission
<uses-permission android:name="android.permission.INTERNET" />



 

Exporting Telerik WPF GridView to CSV, Excel and Word

As everybody knows, Windows 8 is coming very soon and now a wave of developers and companies are making a big effort to implement their products for this new platform with its new style, etc. This desktop applications are based on Windows Presentation Foundation (or WPF) that it is a computer-software graphical subsystem for rendering user interfaces in Windows-based applications. WPF employs XAML, a derivative of XML, to define and link various UI elements. WPF applications can also be deployed as standalone desktop programs, or hosted as an embedded object in a website. WPF aims to unify a number of common user interface elements, such as 2D/3D rendering, fixed and adaptive documents, typography, vector graphics, runtime animation, and pre-rendered media. These elements can then be linked and manipulated based on various events, user interactions, and data bindings. Microsoft Silverlight provides functionality that is mostly a subset of WPF to provide embedded web controls comparable to Adobe Flash. 3D runtime rendering is supported in Silverlight since Silverlight 5 
Now everybody are seeing that with the controls that Windows is giving inside of Visual Studio 2010 are not fitting their needs or to accomplish them they need more time to develop exactly what they exactly need. But, fortunately we have another options that are doing much easier our lives. They are third party controls with an huge extension of choices, as Infragistics,   DevExpress, Mindscape, Telerik, etc.

<Window x:Class="WpfApplication1.Window1"
    Xmlns = "http://schemas.microsoft.com/winfx/2006/xaml/presentation"
    Xmlns:x = "http://schemas.microsoft.com/winfx/2006/xaml"
    Xmlns:telerik = "http://schemas.telerik.com/2008/xaml/presentation"
    Xmlns:local = "clr-namespace:WpfApplication1"
    Title="Window1">
    <Grid>
        <Grid.Resources>
            <ObjectDataProvider x:Key = "Customers" 
                 ObjectType = "{x:Type local:NorthwindDataContext}" MethodName = "get_Customers">
            </ObjectDataProvider>
        </Grid.Resources>
        <Grid.ColumnDefinitions>
            <ColumnDefinition Width="100" />
            <ColumnDefinition Width="100" />
            <ColumnDefinition />
        </Grid.ColumnDefinitions>
        <Grid.RowDefinitions>
            <RowDefinition Height="30" />
            <RowDefinition Height="10" />
            <RowDefinition />
        </Grid.RowDefinitions>
        <ComboBox x:Name="ComboBox1" SelectedIndex="0">
            <ComboBoxItem Content="Excel"/>
            <ComboBoxItem Content="Word"/>
            <ComboBoxItem Content="Csv"/>
        </ComboBox>
        <Button Content="Export" Grid.Column="1" Click="Button_Click" />
        <telerik:RadGridView Name="RadGridView1" ShowGroupPanel="False" 
                             Grid.ColumnSpan="3" Grid.Row="2"
                             ItemsSource="{Binding Source={StaticResource Customers}}" />
    </Grid>
</Window>

After this, we have to write the code to export in the “.cs” file of the Xaml view, with the Button_Click event, that it is calling the method to export the gridview:
private void Button_Click(object sender, RoutedEventArgs e)
{
  string content = "";
  ComboBoxItem comboItem = ComboBox1.SelectedItem as ComboBoxItem;
  string selectedItem = comboItem.Content.ToString();
  if (selectedItem == "Excel")
  {
    extension = "xls";
    content = RadGridView1.ToHtml(true);
  }
  else if (selectedItem == "Word")
  {
    extension = "doc";
    content = RadGridView1.ToHtml(true);
  }
  else if (selectedItem == "Csv")
  {
    extension = "csv";
    content = RadGridView1.ToCsv(true);
  }
  string path = String.Format("Export.{0}", extension);
  if (File.Exists(path)) { File.Delete(path);
  }
  using (FileStream fs = File.Create(path))
  {
    Byte[] info = Encoding.Default.GetBytes(content);
    fs.Write(info, 0, info.Length);
  }
}

As you can see, this process is very simple and easy, but it doesn’t stop here. Telerik have an huge control collections also for another technologies, as well as, Silverlight, Ajax, ASP .NET, Mobilphones and for the most recent Metro Styles of Windows 8 for the new Visual Studio 2012.
I add also the project example to this post to download and see with more detail how I accomplished it.
I hope I have helped you.


How to Download image from Url For Web Scraping

/// <summary> /// Function to download Image from Url
 /// </summary>
 /// URL address to download image
 /// <returns>Image</returns> 
 private Image DownloadImage(string _URL)
 {
        Image _tmpImage = null;
        Application.DoEvents();
        try
        {
                    // Open a connection
                    System.Net.HttpWebRequest _HttpWebRequest = (System.Net.HttpWebRequest)System.Net.HttpWebRequest.Create(_URL);
                    _HttpWebRequest.AllowWriteStreamBuffering = true; 
                    // You can also specify additional header values like the user agent or the referer: (Optional)
                    _HttpWebRequest.UserAgent = "Mozilla/5.0 (Windows; U; Windows NT 5.1; en-US; rv:1.6b) Gecko/20031212 Firebird/0.7+";
                    _HttpWebRequest.Referer = "http://www.google.com/";
                     // set timeout for 20 seconds (Optional)
                    _HttpWebRequest.Timeout = 20000;

                     // Request response:
                     System.Net.WebResponse _WebResponse = _HttpWebRequest.GetResponse();

                     // Open data stream:
                     System.IO.Stream _WebStream = _WebResponse.GetResponseStream();

                     // convert webstream to image
                     _tmpImage = Image.FromStream(_WebStream);

                     // Cleanup
                     _WebResponse.Close();
                     _WebResponse.Close();
            }
            catch (Exception _Exception)
            {
                     // Error
                     Console.WriteLine("Exception caught in process: {0}", _Exception.ToString());
                     return null;
             }
            return _tmpImage;
 }

How to Set Internet explorer proxy from code behind

using Microsoft.Win32;

private void SetIeProxy(string Proxy)
 {
         RegistryKey registry = Registry.CurrentUser.OpenSubKey("Software\\Microsoft\\Windows\\CurrentVersion\\Internet Settings", true);
         registry.SetValue("ProxyServer", Proxy);
         registry.SetValue("ProxyEnable", 1);
 
         // These lines implement the Interface in the beginning of program
         // They cause the OS to refresh the settings, causing IP to realy update
         bool settingsReturn = InternetSetOption(IntPtr.Zero, INTERNET_OPTION_SETTINGS_CHANGED, IntPtr.Zero, 0);
         bool refreshReturn = InternetSetOption(IntPtr.Zero, INTERNET_OPTION_REFRESH, IntPtr.Zero, 0);
 }

Highlight text in webbrowser control on mouse doble click C#

 
private void webBrowser1_DocumentCompleted(object sender, WebBrowserDocumentCompletedEventArgs e)
{
        if (webBrowser1.ReadyState == WebBrowserReadyState.Complete)
        {
               HtmlDocument _document = webBrowser1.Document;
               _document.MouseLeave += new HtmlElementEventHandler(document_MouseLeave);

               IHTMLDocument2 currentDoc = (IHTMLDocument2)webBrowser1.Document.DomDocument;
               HTMLDocumentEvents2_Event iEvent = (mshtml.HTMLDocumentEvents2_Event)currentDoc;
               iEvent.ondblclick += new HTMLDocumentEvents2_ondblclickEventHandler(iEvent_ondblclick);
        }
}
 
 
 
 
 
private void document_MouseLeave(object sender, HtmlElementEventArgs e)
{
         HtmlElement element = e.FromElement;
         mshtml.HTMLElementEvents2_Event iEvent;
         iEvent = element.DomElement as mshtml.HTMLElementEvents2_Event;
         iEvent.ondblclick -=new HTMLElementEvents2_ondblclickEventHandler(iEvent_ondblclick);
}

bool iEvent_ondblclick(IHTMLEventObj pEvtObj)
{
         if (pEvtObj.srcElement.innerText != null)
         {
               //
               // Stop navigation on current click if it contains link.
               //
               if (pEvtObj.srcElement.outerHTML.Contains("href"))
                      webBrowser1.Stop();

               IHTMLDocument2 doc2 = wbMainPage.Document.DomDocument as IHTMLDocument2;
               StringBuilder html = new StringBuilder(doc2.body.outerHTML);
               String substitution = "" + pEvtObj.srcElement.innerText + "";
               html.Replace(pEvtObj.srcElement.outerHTML, substitution);
               doc2.body.innerHTML = html.ToString();
         }
         return false;
} 

Replace string between html tag using regex

 
 
//This method is used to replace the string between html tag by given string including html tag

//There is no need to find index position of StartTag or EndTag. Therefore no chances to raise the any exception

//Replace string if starting and ending is exist

// If you missed end bracket of html tag(>), it will append it self.

public static string ReplaceStringBetweenHTMLTag(string Content, string StartTag, string EndTag, string StrReplace)

{

string StrPattern = string.Empty;

if (StartTag != string.Empty)

{

if (StartTag.Contains(">"))

StrPattern = string.Format("{0}.*?{1}", StartTag, EndTag);

else

StrPattern = string.Format("{0}.*?>.*?{1}", StartTag, EndTag);

Content = Regex.Replace(Content, StrPattern, StrReplace);

}
}
 
Example1:
Content = " Rest of the Content.cboxIE #cboxT After Style type ";
StartTag= ""
EndTag = ""
StrReplace ="blog------- "
Result:
blog------- After Style type

Example2:

Content = "// StartTag= "EndTag = ""
StrReplace ="
blog------- "
Result:
blog------- After Style
 

Friday, December 16, 2011

Data Mining Revenue


Data has been used from time immemorial by various companies to manage their operations.Data is needed by various organizations strategically aimed at expanding their business operations, reduction of costs, improve their marketing force and above all improve profitability. Data mining is aimed at the creation of information assets and uses them to leverage their objectives.
In this article, we discuss some of the common questions asked about the data mining technology. Some of the questions we have addressed include:
-          How can we define data mining?
-          How can data mining affect my organization?
-          How can my business get started with data mining?
Data Mining Defined
Data mining can be regarded as a new concept in the enterprise decision support system, usually abbreviated as DSS. It does more than complementing and interlocking with the DSS capabilities that may involve reporting and query. It can also be used in on-line analytical processing (OLAP), traditional statistical analysis and data visualization. The technology comes up with tables, graphs and reports of the past business history.
We may define data mining as modeling of hidden patterns and discovering data from large volumes of data.It is important to note that data mining is very different from other retrospective technologies because it involves the creation of models. By using this technology, the user can discover patterns and use them to build models without even understanding what you are after. It gives explanation why the past events happened and even predicting what is likely to happen.
Some of the information technologies that can be linked to data mining include neural networks, fuzzy logic, rule induction and genetic algorithms. In this article we do not cover those technologies but focus on how data mining can be used to meet your business needs and you can translate the solutions thereafter into dollars. 
Setting Your Business Solutions and Profits
One of the common questions asked about this technology is; what role can data mining play for my organization? At the start of this article we described some of the opportunities that can be associated with the use of data. Some of those benefits include cost reduction, business expansion, sales and marketing and profitability. In the following paragraphs we look into some of the situations where companies have used data mining to their advantage.
Business Expansion
Equity Financial Limited wanted to expand their customer base and also attract new customers. They used the LoanCheck offer to meet their objectives. Initiating the loan, a customer had to go to any branch of Equity branch and just cash the loan. Equity introduced a $6000 LoanCheck by just mailing the promotion to their existing customers. The equity database was able to track about 400 characteristics of every customer. The characteristics were about loan history of the customer, their active credit cards, current balance on the credit cards and if they could respond to the loan offer. Equity used data mining to shift through 400 customer features and also finding the significant ones. They used the data and build model based on the response to the LoanCheck offer. They then integrated this model to 500,000 potential customers from credit bureau. They then selectively mailed the most potential customers that were determined by the data mining model.At the end of the process they were able to generate a total of $2.1M in extra net income from 15,000 new customers.
Reduction of Operating Costs
Empire is one of the largest insurance companies in the country. In order to compete with other insurance companies, it has to offer quality services and at the same time reducing costs.Therefore it has to attack costs that may in form of fraud and abuse. This demands a considerable investigation skills and use of data management technology. The latter calls for data mining applicationthat can profile every physician in their network based on claims records of every patient in their data warehouse. The application is able to detect subtle deviations on the physician behavior that are linked to her/her peer group. The deviations are then reported to the intelligence and fraud investigators as “suspicion index.” With this effort derived from data mining, the company was able to save $31M, $37M, and $41M in the first three years respectively from frauds.
Sales Effectiveness and Profitability
In this case we look into pharmaceutical sector. Their sales representatives have wide range of assortment tools they use in promoting various products to physicians. Some of the tools include product samples, clinical literature, dinner meetings, golf outings, teleconferences and many more. Therefore getting to know the promotions methods that are ideal for particular physician is of valuable importance and it is likely to cost the company a lot of dollars in sales call and thereby more lost revenue.
Through data mining, a drug maker was able to link eight months of promotional activity based on corresponding sales found in their database. They then used this information to build a predictive model for each physician.The model revealed that for the six promotional alternatives, only three had a significant impact. Then they used the knowledge found in the data mining models and thereby customizing the ROI.
Looking at those two case studies, then ask yourself, was data mining necessary?
Getting Started
All the cases presented above have revealed how data mining was used to yield results to the various businesses. Some of the results led to increased revenue and increased customer base. Others can be regarded as bottom-line improvements that impacted on cost savings and also improved productivity.In the next few paragraphs we try to answer the question; how can my company get started and start realizing the benefits of data mining.
The right time to start your data mining project is now. With the emergence of specializeddata mining companies, starting the process has beensimplified and the costs greatly reduced. Data mining project can offer important insights into the field and also aggregate the idea of creating a data warehouse.
In this article we have addressed some of the common questions regarding data mining, what are the benefits associated with the process and how a company can get started. Now, with this knowledge your company should start with a pilot project and then continue building a data mining capability in your company; to improve profitability, market your products more effectively, expand your business and also reduce costs