Friday 25 January 2019

Process the non 0KB (non Empty) files and load the data into database and filter the 0KB (Empty) files from the source Directory and mail to business

We are getting the bunch of CSV files into the processing directory and we need to load these files into the database. Sometimes the client sends some empty (0kb) files. Our requirement is to load the non-empty files to the database and for the empty files we need to log this information into the table and send this information to the client.
Let’s see how we implement.
Below is the processing directory
 

We have some 0kb files. My requirement is to load Non-0kb files to the database and load 0kb files mail to the client saying that these files are 0kb files, please send it aging these files.

Taking Foreach look container and taking file enumerator
     
     
It will iterate the file one by one.
In the Foreach container, we will take the script task and check the size of the files. If size is greater than 0kb then we will load the data into the table.
 
     
We need to create some variables.
   

Inside the script task, we are getting the file information.
int filesize = 0;
            FileInfo oFileInfo = new FileInfo(Dts.Variables["User::File_path"].Value.ToString());
            Dts.Variables["User::File_Size"].Value = oFileInfo.Length.ToString();
            Dts.Variables["User::File_name"].Value = oFileInfo.Name.ToString();
            filesize = int.Parse(oFileInfo.Length.ToString());
            if (filesize==0)
            {
                Dts.Variables["User::IsEmpty"].Value = true;
            }

Here I am setting the IsEmpty flag. If the file size is 0 kb then the value is true by default value is false.
   

If the file is empty in this case execute SQL task will execute
    

   
In the Execute SQL task, we are inserting the empty file details.
Below is the table
CREATE Table fileInfo
(FileID int identity (1,1),
FileName varchar(50),
FilePath varchar(100),
Filesize varchar(20),
CreatedDate datetime,
CreatedBy varchar(50)
)
Configure the execute sql task and provide the sql statement.
 

Map the parameter.

 
If the file is not empty in this case Data Flow task will execute.
 

In data flow task we will load the data.
 

I am loading the data into the Customer table.
 

Once enumerate all files I am checking any empty file present in the table. I am getting the count from the table.


 If the count is greater than 0 then I am pulling the data from the fileinfo table and send the mail.
  
In the expression, I am checking the count of the error file
 
 

Pulling the data from the Fileinfo table and storing it into the object variable.
 
   
Soring the result in the object variable.
 

Once we get the file info with the help of script task I send mail.
  

Below is the code for sending the mail in HTML format
string smtpAddress = "smtp.gmail.com";
             int portNumber = 587;
             bool enableSSL = true;
             string emailFromAddress = "****************@gmail.com"; //Sender Email Address 
             string password = "*******"; //Sender Password 
             string emailToAddress = "*****************@gmail.com"; //Receiver Email Address 
             string subject = "Empty Files";
             string body = "Hello, Below files are empty. Send it again.";
            DataTable dt = new DataTable();
            var oleDa = new OleDbDataAdapter();
            oleDa.Fill(dt, Dts.Variables["User::FileInfo"].Value);

            using (MailMessage mail = new MailMessage())
            {
                mail.From = new MailAddress(emailFromAddress);
                mail.To.Add(emailToAddress);
                mail.Subject = subject;
                mail.Body = body;
                mail.IsBodyHtml = true;

                mail.Body += " <html>";
                mail.Body += "<body>";
                mail.Body += "<table>";
                mail.Body += "<tr>";
                mail.Body += "<td>FileName</td><td>FilePath </td><td>Filesize</td><td>CreatedDate</td>";
                mail.Body += "</tr>";
               foreach (DataRow row in dt.Rows)
                {   mail.Body += "<tr>";
                    mail.Body += "<td>" +row.ItemArray[0].ToString() + "</td><td>"+ row.ItemArray[1].ToString() + "</td><td>"+ row.ItemArray[2].ToString() + "</td><td>"+ row.ItemArray[3].ToString() + "</td>";
                    mail.Body += "</tr>";                 
                }              
                mail.Body += "</table>";
                mail.Body += "</body>";
                mail.Body += "</html>";
                using (SmtpClient smtp = new SmtpClient(smtpAddress, portNumber))
                {
                    smtp.Credentials = new NetworkCredential(emailFromAddress, password);
                    smtp.EnableSsl = enableSSL;
                    smtp.Send(mail);
                }

Now Package is ready to run.
                             

I am executing this package.
                             
OOOOOOOOOOO my god, send mail failed.
It may be the cause of below
Case 1: when the password is wrong
Case 2: when we try to log in from some App
Case 3: when we try to log in from the domain other than your time zone/domain/computer (This is the case in most of the scenarios when sending mail from code)

I need to set less secure app access on
   
Now I am turning on
Now I am running this package.


Package executed successfully. See the mail


See here we get the expected result.

Load data from multiple Excel sheets (fixed number of sheet with same Meta data) into the database

I have excel file in that I have a fixed number of sheets (3 sheets example ) with same Metadata. This file I am getting daily we need to load this data into the database.
See the example file
  
In this excel file, I have 3 sheets named as 1000 records, 100 Records, 5000 Records having 1000,100,5000 records data.
I need to load this data.
These are multiple ways to load this data. Here I will see the simple demo how we load these data in a simple way.
Let’s see the demo.
I am taking the data flow task
                               
In data flow, I am taking source as excel
  
Now I am creating an excel connection manager and configuring the source file.
   
Now double click on the Excel source component.
In the connection manager tab, we need to select the connection manager and select SQL command as Data access Mobe

In SQL command text, write the sql command to pull the records form all sheets.
  

Keep in Mind: It will pull the records from the provided sheets only. For example, I wrote sql script to pull the records from 3 sheets only if in the excel file have one more sheet, in this case, it will put 3 sheets (as above written in the SQL command) only.
Click ok.
Now taking destination as OLEDB Destination
  
And doing the configuration.
Now the package is ready to run.
See the records in the table.
  
Now running this package.
I am expecting 6100 records load into the database
  
Now see in the database

  
Got the Expected result.

Thursday 24 January 2019

Load only first 10 records from the excel into the database table

I am getting an excel sheet in this sheet there is n number of records but our requirement is to load only first 10 records from this excel sheet.
In this demo, we will see how to load only first 10 records from the excel.
I am using below excel file which has 1000 records.

I am taking a Data flow task.
   
In Data flow, I am taking Source as excel
   
Creating an excel connection.
  
Click ok. Select Data access mode as SQL Command.

Now we need to write the SQL query to pull the data from excel sheet.
  
Select * from [1000 Records$A1:AK11]
Click on the Preview button in review window we will see only 10 records.
If we want to read the whole sheet then we can write SQL query as below
Select * from [1000 Records$]
But we want to get the data from the specified range so we will provide the range.
Click ok.
Now I am taking destination as OLEDB and doing the mapping.
  
Before running this package records in the database.
  
Now running this package.

See only 10 records are loaded.
Let’s see the records in the database.



Get the expected result.

Skip n rows from excel and load the data into the database using SSIS

My requirement is to load the excel data into the database. I am getting the excel file from the business in that excel sheet first 5 rows having header and site details. Data starts from the 6th row so we need to read the data from 6th rows onwards. Below is a simple example file.
  

 Let’s see it in this demo
Taking Dataflow task  
In Data flow, I am taking Source as excel
  
Creating an excel connection.
  
Click ok. Select Data access mode as SQL Command.
  
Now we need to write the sql query to pull the data from excel sheet.
  
Select * from [100 Records$A6:AK10000]
If we want to read the whole sheet then we can write SQL query as below
Select * from [100 Records$]
But we want to get the data from the specified range so we will provide the range.
Click ok.
Now I am taking destination as OLEDB and doing the mapping.
  
Before running this package records in the database.
  
Now running this package.
  
100 rows loaded successfully.
Records in the database.


Popular Posts