How to check Duplicates based on Multiple Fields – SQL or SAS

You have big data sets and that contains duplicate but you need to check duplicates based on combination of two or three columns. How can you check that using SQL or SAS? So, you can use the following combination to check duplicates based on multiple fields or columns in SQL or SAS:

Query: There is a dataset called “Country_Polulation” with following fields: 1. Name 2. Age 3.CityName 4.DOB 5.Year

You need to check duplicate records – like if Name and DOB of two records is same that means that is a duplicate. Follow the following methods to check duplicates based on multiple fields:

Let’s say we are checking duplicates for year – 2018

Using Having Clause:

Having clause will check the count of name and DOB if exists more than once, that name will be exported from inner query to export query. If you need to learn basics of SAS, follow these articles: SQL Basics and Learn Basic Functions of SQL

Select A.name, A.DOB

from Country_Polulation as A

where name in

(

Select B.name, B.DOB

from Country_Polulation as B

where year=2018

Order by name

Group by Name, DOB

Having (count(name)>1 and count(DOB)>1)

)

order by A.Name

 

2. Check Duplicates in SAS:

You will have to use Proc step for this task if you want to retrieve all the duplicate records. Basically do a self join and check all the fields with each other for equality to check duplicates.

Proc SQL;

Create table Dupe_Records as

Select A.Name, A.DOB, B.*

from Country_Polulation as A

JOIN

( Select B.Name, B.DOB, Count(*) as Quantity

from Country_Polulation  as B

where year=2018

group by B.name, B.Dob

having count(*)>1

)

order by Name, DOB

This Dupe_Records table will contain all the duplicate records ordered by name and DOB in ascending order. The reason why we couldn’t mention Name and DOB condition in Having clause in SAS is – SAS deals differently with Dates. and DOB is a Datetime18. format here when I tried the same.

3. Check Duplicates in Excel:

If the dataset is of small size, you can follow the mentioned steps:

  • Using Home Tab:

Home Tab -> Click on Conditional Formatting -> Click on Highlight Cell Rules -> Duplicate Rows -> Check the colors to highlight the duplicate text

  • Using Data Tab:

Select the data range or the columns wherein you need to check duplicates -> Data Tab -> Click on Remove Duplicates in Data Tools

This will give you the option of selecting columns to consider dupes. Select the desired columns and the dupes will be removed.

Hope this post will help you 🙂

Date Time Functions in SQL and the Use of Date Time Functions

Date Time functions in SQL

There are various Date time functions available in SQL which allow us to create trigger based on it. Date time function allow us to check whether an input string is a date or not. Also, we can check what’s the day or week number on a specific date using date time function.

Date Time

Let’s check various date time function available with us:

  1. IsDate

The isDate function is used to check whether the input string is a date or not.

Command: Select isDate(“2012-05-30”), it will return 1

If the input string is not a date, the IsDate date time function will return zero.

The isDate function works well on date, time and datetime datatype but not with datetime2 (timestamp with nanoseconds)

2. Day()

This date time function returns the “day number” of the specified input date string.

Command: Select Day(“2012-02-30 02:35:35”) – Returns 30

3. Month()

This sql date time function returns the “month number” of the specified input date string.

Command: Select Month(“2012-02-30 02:35:35”) – Returns 2

4. Day()

This sql date time function returns the “day number” of the specified input date string.

Command: Select Day(“2012-02-30 02:35:35”) – Returns 30

5. DateName()

The DateName() date time function returns a string value while other date time functions return integer value.

There are various attributes that can be used with the DateName() function. let’s me share some examples of it here:

  1. DateTime(Day, “Date value”|”DateTime Value”) – This function returns the day value (exception it is integer)

Command: Select DateTime(Day, “2013-05-23”) – the output will be 2

2. DateTime(WeekDay, “Date value”|”DateTime Value” – This function returns the week day value

Command: Select DateTime(WeekDay, “2013-05-23”) – the output will be Saturday.

3. DateTime(Month, “Date value”|”DateTime Value”)  – It returns the month value as a string

Command: Select DateTime(Month, “2013-05-23”) – the output will be May.

6. Datepart()

The DatePart() function works similar way as DateName function to find out the weekDay. the only difference between DatePart() and DateName() function is Datepart() returns an integer.

e.g. Select DatePart(WeekDay, “2013-05-25”) will return 2 if it is Tuesday

7. DateAdd()

This sql function will add day, month or year according to the specified value.

Command: Select DateAdd(Day|Month|Year, integer value of addition, Date Value)

e.g. Select DateAdd(Day, 10, “2013-05-20”) – This will output “2013-05-30”

Date Diff

8. DateDiff()

This sql function will add day, month or year according to the specified value.

Command: Select DateDiff(Day|Month|Year, Date value 1,Date value 2)

e.g. Select DateAdd(Month, “2013-05-20″,”2013-01-20”) – This will output 4

Cast and Convert Functions in SQL

Cast and convert functions allow you to convert one data type into another.

Syntax of Cast function:

Select cast(Column_name as Data type)

e.g. Select cast(date as nvarchar)

Syntax of Convert function:

Convert function lets you choose style parameter for string values only. The style feature is not available in cast function. By styling I mean a user defined format in which the output is required.

e.g. Select convert(data type, column_name, style)

There are some specific style codes like 103 means : dd/mm/yyyy. Thus the following example:

Select convert(nvarchar, DateOfBirth, 103)

Stored Procedures and String Functions in SQL

Hola friends, let’s understand what are stored procedures and the benefits of using it. I hope you have learnt some basics of SQL before this. Learn basics of SQL in my previous post.

What are Stored Procedures?

Stored procedures are a set of instructions that need to be instructed again and again. In SQL we can save this frequent instructions as a procedure and call them by just their name.

 

Stored Procedures

How to create Stored Procedures?

To create stored procedures, simply use the Create Proc/Procedure Procedure_name command.

We can also pass parameter in a stored procedure. The parameters can be of two type:

1. Input (Used to take input)

2. Output (used to print output) – The output keyword must be specified as output in the description.

One example of the stored procedure is shown in the below image:

 

Stored-Procedure1

Running the Stored Procedures:

The command used to execute a stored procedure is: EXEC|Execute Procedure_name

e.g. Exec HumanResouces.uspFindEmployee “123” (Here 123 is the value for the @BusinessEntityID input parameter.

Stored Procedure with Output Parameter

An example of output parameter is:

Create Proc spGetEmployeeCount

@LastName int

@EmployeeCount Output

As

Begin

Select @EmployeeCount =Count(*) from emp where LastName = @LastName

End

Executing Stored Procedure with Output Parameter:

To execute the stored procedures with output parameters, it is very important to declare the output variable first. To declare the output variable, use the following command:

Declare @OutputVariableName Datatype

Exec Storedprocedurename Input parameter, Output Parameter Out|Output

Print@OutputVariableName

e.g.

Declare @TotalCount Int

Exec spGetEmployeeCount @LastName = “Male”, @TotalCount Out

Print@TotalCount

Benefits of Stored procedures:

  1. The execution plan can be reused

When a normal statement is executed, the path like first few columns are selected, then refined through where clause and then ordered. This is called a kind of execution plan. So, we can save time through stored procedures.

e.g. Select name, gender from Emp

when id IN(12, 14, 15)

order by name

A normal statement if executed with different parameters, use different execution path. However Stored procedures uses the same execution path even with different variable values.

2. Reduced Network Traffic

Since Stored procedures allows code re-usability, the sql instructions takes less time and space while utilizing network bandwidth. Thus network traffic is reduced.

3. Easy Maintainability using Stored Procedures

Maintenance becomes easy. A change in stored procedure is easy rather than finding similar statements at various places and then modifying the code.

4. Code Re-usability

The instruction that will be required again and again need not to be typed again. We just have to create a stored procedure with that statement and this will help us in reusing the code in less time and space.

5. Better Security

The database can be huge and we don’t want everyone to give access to every user in the network. So, build procedures on the specific table to which the user wants access. This way we will be able to provide better security.

6. Avoid SQL Injection Attack

Stored procedures also helps in avoiding SQL injection attacks. To know more about SQL injection attacks, go to the link.

Learn basics of SQL here

String Functions in SQL

  1. Left function:

This string function in sql returns integer values from the left.

command: select left(string, integer count)

e.g. Select left(“abcd”, 3) – Result will be “abc”

2. Right Function

This string function in sql returns integer values from the right of a string/column value.

command: select left(string/column, integer count)

e.g. Select left(“abcd”, 3) – Result will be “bcd”

3. CharIndex

This sql function returns the index of the character value.

command: Select charindex(“a”, “abcd”) – answer would be 1

4. Len

This function the total length of a string or column value of string type.

command e.g. : Select len(“abcd”) – answer would be 4

Note: This sql string function will not count the blank values at the end of a string.

5. Substr

This sql function is used to select a substring value from a string.

Command: select substr(“string value”|column name, position to start, length of fetch character

e.g. Select substr(“abcd”, 1, 3) – answer would be “abc”

We can also choose negative indexing here. A -1 indicates from the right.

6. Replicate

Replicate function replicates a specific string to the specified number of times.

command: Select replicate(“string”, number of times to replicate)

e.g. Select lastname + replicate(“*”,5)

This command will repeat the * five times in lastname. Let’s say the lastname is “John” then the output is: “John*****”

6. Space

This sql function will insert space to the specified number of times between column values.

command: Select Lastname + Space(5) +First name – This command will introduce a 5 character space between first name column value and last name.

7. PatIndex (or Pattern Index)

PatIndex works the same way as charindex by telling the first occurrence. However, PatIndex allows you to use wildcard. You can’t use wildcard with CharIndex.

e.g. Select PatIndex(“aaab”, “abcaaababc”) – The answer would be 4

If it do not find any matched string, the sql string function returns zero.

8. Replace

This string function replaces a string into another one.

Command: Select(String, “value to replace”, replacement value”)

e.g. Select(“abcd.com”, “.com”, “.net”) – The result will be “abcd.net”

9. LTRIM

This string function is used to trim leading blanks from the beginning.

Command e.g. : Select LTRIM(LastName)

10. RTRIM:

This sql function trims the trailing blank values.

e.g. Last Name : “abcd  ”

Select RTRIM(LastName) – the result will be “abcd”

11. ASCII:

This SQL function returns the ASCII code of the string value.

Command: Select ASCII(string value)

e.g. Select ASCII(A) – The answer would be 65

12. CHAR

This SQL function returns the character value from the integer ASCII code.

Command: Select CHAR(Integer value)

e.g. Select CHAR(65) – The answer would be A

13. STUFF

This sql function replaces a specific string value at the specified position by replacing the characters up to length value. It is kind of masking.

Command: Select STUFF(Column, Starting position, length, String to be used as replacement)

e.g. Select STUFF(LastName, 1, 3, “**”)

If the last name is “JohnMarcel”, the output of stuff function will be “***nMarcel”

SQL Basics and Queries : SQL Tutorials

What is a Database?

Database is an organized collection of related information. In daily world we deal with lots of data. In this internet technology more and more data is getting produced these days. We have multiple database management systems available with us to manage, store and update this enormous data in convenient way. e.g. Oracle, Sybase, Microsoft SQL server management studio etc.

DBMS and SQL

DBMS and SQL

DBMS (Database management server) is a collection of software tools used to manage, update, retrieve the data from the database. SQL (Structured query language is used to connect the DBMS with the database.

DBMS

All queries have been executed on the Microsoft SQL Server management Studio version 17.0. SSMS is a client tool and not the server. It is rather used as a tool to connect to the database server.

Settings: Local Host

Connect: Database Engine

Use SQL authentication username and password

SQL Databases:

In Microsoft SQL Server Management Studio you will find two types of databases:

  1. System Database
  2. User created Database

-System database can’t be deleted

SQL Command Types:
  1. DDL (Data Definition language) – Used to define/create database object
  2. DML (Data manipulation language) – Used to insert values, update and delete values from the database object created by DDL commands.
  3. TCL (Transaction Control language) – Used to control transactions through Commit and Rollback commands

SQL DDL Commands – data definition language (Create, Alter and Drop commands)

1. Creating a database:

Database can be created either using GUI or through SQL query in SSMS.

Create statement is used for this purpose: Create [Database Object] [Database Object name]

Ex. Create Database db1 (this statement will create a database with name db1)

Whenever we create a database, two types of files are created with it: 1. .MDF file (contains actual data) 2. .LDF file (contains log file)

2. Modify a Database:

Alter statement is used to alter a sql database object.

Alter Command:  Alter [Database Object] [Database Object name] Modify Col1 = Value

E.g. Alter Database db1 Modify Name = db2 (this will change the name of the database)

Renaming through stored procedure: sp_renameDB [Old database name] [New database name]

e.g. sp_renameDB db1 db2

3. Dropping a Database:

Drop statement is used to delete a database completely from the system(.mdf and .ldf files are also deleted with it)

Drop command: Drop [Database Object] [Database Object name]

e.g. Drop Database db1 (this will delete database db1)

Note- If a database is getting used by any other user, make sure that database is not getting used by any other database. Else an error will be generated

Resolve this single user thing, use this command:

Alter Database db1 set Single_USER with Rollback immediate

(Rollback immediate, rollback any commands and delete the database immediately)

SQL DML Queries : Insert, Update, delete

1. Create a Table:

Command: Create Table [table name] ([column name] [data type of column] [constraint])

e.g. Create table t1(ID int NOT NULL Primary Key, Gender nvarchar(20) NOT NULL)

This command will create a table with name t1 and 2 columns ID and Gender of INT and nvarchar datatypes respectively. nvarchar is a UNICODE data type and store 2 bytes per characters, while varchar stores 1 byte per character.

In order to store the table in a particular database use the following command:

Use [database name]

Go

Create table command….

Primary Key – Can’t be null and must be unique. It uniquely identify each row in the table

Foreign key– It can contain null values and it references primary key present in some other values (basically the column in which it looks for a value). Foreign key is used to establish relationship between two tables. It is used to enforce database integrity.

Create a Foreign key relation –

Alter table [table name] add constraint [constrain name] foreign key(foreign key column name) references [PrimaryColumn Table Name] (primary key column)

e.g. Alter table tb1 add constraint tb1_genderid foreign key(tb1) references tb(id)

Note- Constraint name should make sense like tablename_columnName

2. Select all values of a table:

Command: Select * from [table name]

To select all tables of a database choose:

Select * from dual (dual refers data dictionary)

3. Insert values in a table :

Insert command is used to insert values in a table: Insert into [table name] (col 1, col 2, …) Values(col 1 value, col 2 value,…)

e.g. Insert into a1(id, name, gender) values(11, “ss”, “male”)

4. Adding a Default value in a column:

We can assign default values to a column rather than assigning Null values:

Alter table [table_name] add constraint constraint_name Default [default value] For [column name]

e.g. Alter table tb1 add constraint tb1_gender default 2 for gender

This command will assign default value 2 to column gender if value not explicitly defined.

5. Adding a New column into table:

Command: Alter table [table name] add [column name] [column data type] [NULL|NOT NULL] add constraint [constraint name] Constraint

Alter table tb1 add Address nvarchar Not Null add constraint tb1_address default ‘xyz’

This command will add one column Address to the table tb1 that don’t accept null value. Also, default value of ‘xyz’ will be assigned to it.

6. Dropping a Constraint:

Command: Alter table [table name] Drop Constraint [constraint name]

e.g. Alter table tb1 drop constraint tb1_gender

This will drop the constraint.

7. Delete a Table record:

To delete a table record, we use delete command:

Delete from [table name] where column1=”column value”

Note: Where clause is used to put some condition on search selection

However, you can’t delete a table record if the table is getting used by some other user. There are some cascading referential integrity constraint imposed on foreign key constraints.

8. Cascade Referential Integrity Constraint:

We can choose options if a foreign key constraint reference is getting deleted. Four options are there:

  1. No Action : This will simply generate an error if a record from primary key table is deleted that has some value in foreign key table.
  2. Cascade: This option will delete all the foreign key records that are related to primary key will be deleted
  3. Set NULL: This option will set the foreign key dependent value to Null.
  4. Set Default: This option will set the foreign key dependent value to default values provided to the column.

9. Adding a Check Constraint:

This constraint is used to enforce value checks on column. For e.g. The value in the age column>4

Command: Alter table [table name] add constraint [constraint name] check (boolean expression)

e.g. Alter table tb1 add constraint tb1_age_check check(AGE>0 AND AGE<30)

This command will only let you add age between 0 and 30 in the Age column.

Note: The check constraint returns a Boolean value based on which the value is entered in the table. It also let you insert Null values because for Null values, check constraint returns “Unknown”.

10. Identity Column:

It is a column property in SSMS.

Identity column is a column to which values are automatically assigned.

Identity Seed: A value with which the identity column value starts

Identity Increment: The value with which identity column value is incremented.

Command: Create table stu(id int identity(1,1) Primary key)

This command will create a stu table having id as a identity column. The id column here will start from 1 and incremented by 1.

 

10. Setting up External Values/Explicit Value to Identity Column:

To set up external value in Identity column, add the following command before inserting values in table:

Command: Set IDENTITY_INSERT [table name] ON

Insert into table name(column list) values(1,”23″..etc)

11. Setting Off External Values/Explicit Value to Identity Column:

Command: Set IDENTITY_INSERT [table name] OFF

Insert into table name(column list) values(1,”23″..etc)

Note: To reset the identity column value, use DBCC command.

12. Unique Key Constraint:

Unique key constraint is used to enforce unique values in database. There is a slight difference between primary key and unique key.

Primary key values = Unique+Not Null

Unique constraint value = Unique + values can be null

Command: Alter table table_name add constraint constraint_name unique(column name)

or

Create table Stu(Name varchar(20) Unique)

13. Applying a Trigger:

Firstly let’s try to understand what a trigger is. A trigger is an sql instruction/set of instructions that will will cause an action once a specific condition occurs. For example: Inserting another row table 2 when a row is entered in the table 1.

Command:

Create Trigger [trigger_name] on [table_name] for Insert/Update/Delete/Condition

as

begin

[instructions]

end

14. Selecting values from table:

Select is a command used to retrieve records from a table.

  1. To fetch all records from a table:

Command: Select * from [table_name]

e.g. Select * from emp

2. Select specific columns from a table:

Command: Select [col_name_1], [col_name_2]… from [table_name]

e.g. Select name, age, id from Employee

3. Fetch all distinct records from a table:

Command: Select distinct [column_name] from [table_name]

e.g. Select distinct name from Employee

This command will help in fetching the distinct records from table Employee by Name column.

4. Fetch record matching a specific condition:

Where is used to apply a specific condition in the SQL command.

Command: Select * from table_name where column_name = condition value

e.g. Select name, id from employee where name=”John”

This command will fetch all the records from the table with name column value as john.

5. Fetch record not matching a specific condition (column value):

Command: Select * from table_name where col_name <> Column value

“<>” signifies as not equal to here. We can also use “!=” to compare values.

6. OR operator in SQL:

OR operator is used to specify two or more conditions together.

Command: Select * from table_name where col1=value OR col2=value

e.g. Select name, age, salary from Employee where name=”John” OR name=”Nick”

This sql command will fetch all the table records where name is either John or Nick

7. AND Operator in SQL:

AND operator is used to specify two and more conditions together.

Command: Select * from table_name where col1=value AND col2=value

e.g. Select name, age, salary from Employee where name=”John” AND age=”30″

This sql command will fetch all the table records where name is John and age is Nick

8. IN Operator in SQL:

IN operator is used to retrieve records where condition matches more than 1 value. (And you don’t want to use OR multiple times in a sql command)

Command: Select * from table_name where col_name IN(value1, value2, value3…)

e.g. Select * from Employee where age(21, 25, 30)

This command will fetch all the table record where age is either 21 or 25 or 30.

SQL Wildcards

SQL supports various kind of wild card characters to facilitate data retrieval in multiple ways. Please refer the image for all sql wild card characters.