Friday 30 September 2022

Load the data from SQL server to SNOWFLAKE table using SSIS

We can load the SQL server table data into the SNOWFLAKE using SSIS. To load the data we are using the ODBC destination to connect the SNOWFLAKE and loading it.

Before creating the package we need to create the ODBC DSN.

Read here: Creating ODBC Connection for the SNOWFLAKE

Taking Data flow task in SSIS.  

Now taking ODEDB Source to get the data from the SQL server table. 

Now taking ODBC Destination.

Before that we are creating the ODBC connection.

Right click on the connection manager pan  

Click on the new connection and select the ODBC Connection and click on the add button.

Read here: Creating ODBC Connection for the SNOWFLAKE

https://bageshkumarbagi-msbi.blogspot.com/2022/09/creating-odbc-connection-for-snowflake.html

Select the DSN name and provide the SNOWFLAKE User name and password and click on the Test Connection button to test the connection. 

Click Ok. ODBC Connection is created successfully.

Now taking Destination as ODBC destination and selecting connect as ODBC.  

Providing the table name  

Doing the column mapping.  

Click ok.

Our package is ready to run.

See the source data. 

 See the destination table 

                         

Now running the package.

Package executed successfully.    

Now see the records into the SNOWFLAKE table.  

1k rows loaded successfully.

Creating ODBC Connection for the SNOWFLAKE

 When we are creating the DSN for ODBC connection to connect the Snowflake, First time we need to check that snowflake odbc driver install in our system or not.  Without this driver we can’t create the DSN.

First we need to install the Driver. Below is the official website to download the driver.

https://sfc-repo.snowflakecomputing.com/odbc/win64/latest/index.html    

After downloading the driver we need to install this.

Let’s create the DSN

Open ODBC Data Source Administrator and select User DSN and click on the Add button.    

Select SnowflakeDSIIDriver  

Click finish button                

Here we need to provide all information. After filling all details click on the Test button.  

Now click ok.


DSN Created successfully.

Saturday 13 August 2022

AlwaysCheckForRowDelimiters in SSIS

We have a file as below                      

  

We need to load this file into the database. See in this file we are getting irregular number of columns. While we are loading the data into the database it is not loaded in the correct format.

See this demo

We have below table 

Now we are taking source as flat file source and oledb destination and doing the mapping. 

Now we are running this package.   

Package executed successfully. It load 3 records only but in the file there are 5 rows.

Let’s see the records in the table.

 

It is not loaded in correct format.

In this case, what needs to be done is:-

SET AlwaysCheckForRowDelimiters Property of connection manager to true (If it's not set by default) and we will see all rows behaving normally when we do the preview. 

By default, the flat file connection manager always checks for a row delimiter in unquoted data and starts a new row when a row delimiter is found. This enables connection manager to correctly parse files with rows that are missing column fields.

In some cases, disabling this feature may improve the package performance. You can disable this feature by setting the flat file connection manager property, AlwaysCheckForRowDelimiters, to False.

Currently AlwaysCheckForRowDelimiters (see the file connection manager property) is false due to that data is not loaded properly.

 

Now we are setting it true. 

Now we need to run this package.

Before running the package see the data in the table. 

Now running the package. 

Package executed successfully and 5 records are loaded into the table. 

See those column don’t have value for that the value will be null. 

AlwaysUseDefaultCodePage property in OLEDB Source in SSIS

Recently we are getting the below error.

Data Flow Task: The column “ABCD” cannot be processed because more than one code page (XXXX and YYYY) are specified for it.

By default  AlwaysUseDefaultCodePage property value is false. 

It means the conversion from Unicode (65001) to ANSI (1252) is not done automatically within the data source step.

By setting it true it will done automatically. 


Thursday 14 July 2022

DTUTIL exit codes

 The Dtutil command prompt utility is used to manage SQL Server Integration Services packages. The utility can copy, move, delete, or verify the existence of a package. These actions can be performed on any SSIS package that is stored in one of three locations:

·         Microsoft SQL Server database

·         SSIS Package Store, and

·         File system

Read more here:

 https://bageshkumarbagi-msbi.blogspot.com/2018/12/dtutil-utility-command-in-ssis.html

We can run DTUTIL from other applications and it will return some codes to show what happened after running the command; these are DTUTIL exit codes.

 

                Value

Description

0

The utility executed successfully.

1

The utility failed.

4

The utility cannot locate the requested package.

5

The utility cannot load the requested package.

6

The utility cannot resolve the command line because it contains either syntactic or semantic errors.

 

Tuesday 21 June 2022

This SP is failing in the database when we are running it manually but when we are calling this SP in the package our package is executed successfully

Recently we get a production issue. Our issue is that In the SSIS package using execute SQL task we are calling the SP. This SP is failing in the database when we are running it manually but when we are calling this SP in the package our package is executed successfully. Ideally, our package needs to be failed.

Let’s replicate this issue after that we will see the solution to fix this issue.

We are creating an SP that inserts the record in the table.

Table scrip

create table emp

(id int identity(1,1) primary key,

emp_name varchar(50),

emp_add varchar(50),

emp_mob varchar(10)

)

--Log table

create table Error_log

(

log_id int identity(1,1) primary key,

Error_msg varchar(max)

)

Creating SP to insert the record.

CREATE PROCEDURE P_insert_emp

                (

                @emp_name varchar(50),

    @emp_add varchar(50),

    @emp_mob varchar(100)

                )

AS

BEGIN

declare @rt int;

set @rt=1;

begin try

insert into emp (emp_name,emp_add,emp_mob)values (@emp_name,@emp_add,@emp_mob);

end try

begin catch

set @rt=-1;

insert into Error_log(error_msg) select ERROR_MESSAGE();

end catch;

return @rt;

END;

 

Now executing the Sp

declare @result int;

execute @result= P_insert_emp 'Bagesh','Noida','88888XXXXX';

select @result;

 Record inserted successfully.  

Now calling this SP in the Package

Taking Execute SQL task and doing the configuration.  

Now running the package.  

Executed successfully.  See the records in the table. 

Now we are producing the error in the SP.

declare @result int;

 execute @result= P_insert_emp 'Bagesh','Noida','88888XXXXX678905';

Passing the emp_mob value more than ten characters.  It is throwing an error which we have logged in the Error_log table.  

Now we are calling this SP with the same parameter in SSIS package.  

Now we are executing this package.  

Package executed successfully. Even though our SP is failing.

See the records in the table.  

We need to fail the package if the SP is failed in the DB.

To fail the package we will we the raiserror Function.

To use this function our SP must have either return statement or Output parameter then only we can fail our package.

In this demo SP we have return statement.

Writing below script in the Execute SQL task.

declare @result int,

        @error_msg varchar(max); 

execute @result= P_insert_emp 'Bagesh','Noida','88888XXXXX678905';

 if (@result=-1)

begin

set @error_msg= (select Error_msg from Error_log where log_id=(select @@IDENTITY));

 set @error_msg =' Error while inserting records in the Emp table. Error message :' + @error_msg;

 raiserror(@error_msg , 16, 1);

end;


  

Now we are running this package.

  

Package executed successfully.

See the error.

[Execute SQL Task] Error: Executing the query "declare @result int,

        @error_msg varchar(ma..." failed with the following error: "Error while inserting records in the Emp table. Error message :String or binary data would be truncated in table 'Test_DB.dbo.emp', column 'emp_mob'. Truncated value: '88888XXXXX'.". Possible failure reasons: Problems with the query, "ResultSet" property not set correctly, parameters not set correctly, or connection not established correctly.

 This is the best way to use the raise error function when we are calling the SP or function in the Execute SQL task in the SSIS package. 

Sunday 29 May 2022

Difference between Dimension tables and Fact table

 Below is the difference between them

Parameters

Fact Table

Dimension Table

Definition

Measurements, metrics or facts about a business process.

Companion table to the fact table contains descriptive attributes to be used as query constraining.

Characteristic

Located at the center of a star or snowflake schema and surrounded by dimensions.

Connected to the fact table and located at the edges of the star or snowflake schema

Design

Defined by their grain or its most atomic level.

Should be wordy, descriptive, complete, and quality assured.

Task

Fact table is a measurable event for which dimension table data is collected and is used for analysis and reporting.

Collection of reference information about a business.

Type of Data

Facts tables could contain information like sales against a set of dimensions like Product and Date.

Evert dimension table contains attributes which describe the details of the dimension. E.g., Product dimensions can contain Product ID, Product Category, etc.

Key

Primary Key in fact is mapped as foreign keys to Dimensions.

Foreign key to the facts table

Storage

Helps to store report labels and filter domain values in dimension tables.

Load detailed atomic data into dimensional structures.

Hierarchy

Does not contain Hierarchy

Contains Hierarchies. For example Location could contain, country, pin code, state, city, etc.

 

Popular Posts