Need to remove all Dup lines

UNIX Programming, Certification, System Administration, Performance Tuning Reference Books Need to remove all Dup lines

I have a small problem and have no idea how to complete it. I have a script that is colleting all the info I need script below.
*****************************************
#!/usr/bin/ksh
TODAY=$(date +%m%d%y)

ls /export/home/uprdjde1/scripts/JDE_* > tmp.txt
VAR1=tmp.txt

while txtline=$(line)
do
echo "$txtline" |cut -d"/" -f6 | read DIR

egrep "Step|/export/home/uprdjde1/scripts" "$txtline" > tmp1.txt
VAR3=tmp1.txt

while txtline2=$(line)
do
echo "$txtline2" |cut -d" " -f1 | read INPUT
echo "$txtline2" |cut -d" " -f2 | read JOB1
echo "$txtline2" |cut -d" " -f5 | read JOB2
echo "$txtline2" |cut -d" " -f6 | read JOB3
echo "$txtline2" |cut -d" " -f3 | read JOB4
if [ "$INPUT" = "/export/home/uprdjde1/scripts/runube.scr" ]
then
echo "$JOB2" >> JDE_RunList.txt
elif [ "$INPUT" = "/export/home/uprdjde1/scripts/jde_rc.pl" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "/export/home/uprdjde1/scripts/sqlplus_mail.scr" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "/export/home/uprdjde1/scripts/sqlplus.scr" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "/export/home/uprdjde1/scripts/filewatch.pl" ]
then
JOB1=`echo "$JOB1" | tr "[/]" "[ ]"`
echo "$JOB1" |cut -d " " -f5 | read TRIGGER
echo "TriggerJob" >> JDE_RunList.txt
elif [ "$INPUT" = "/export/home/uprdjde1/scripts/sqlloader.scr" ]
then
echo "SQLLoader" >> JDE_RunList.txt
elif [ "$INPUT" = "/export/home/uprdjde1/scripts/jde_rc.pl" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "date" ]
then
echo "$txtline2" |cut -d"-" -f2 | read TITLE
echo "$TITLE:" >> JDE_RunList.txt
fi
done < $VAR3
done < $VAR1
\rm tmp.txt
\rm tmp1.txt

awk '{printf;getline;print " " $0}' JDE_RunList.txt > JDE_RunList.out
VAR4=JDE_RunList.out

while txtline3=$(line)
do
echo "$txtline3" |cut -d":" -f1 | read SWITCH1
echo "$txtline3" |cut -d":" -f2 | read SWITCH2
printf '%-12s%-48s%-9s%-10s%-19s\n' "$SWITCH2" "$SWITCH1" "c.JDE " "cProcess " "device.cProcess" >> JDE_JobList.txt
done < $VAR4
sort JDE_JobList.txt > JDE_JobList1.txt
cat JDE_JobList1.txt | tr "[\"]" "[ ]" > JDE_JobList.txt

\rm JDE_RunList.out
\rm JDE_RunList.txt
****************************************
Here is what my first couple lines look like.
R007011 Unposted Batches
R007011 Unposted Batches
R007021 Transactions to Batch Headers
R007021 Transactions to Batch Headers
R007031 Batch to Detail and Out of Balance (Updates)
R007031 Batch to Detail and Out of Balance (Updates)
R03B16 Statistics History Update
R03B16 Statistics History Update

The next step I need to do is get ride of any duplicate lines. Also if you see anything I should change in my script to make it work better or fast or something I could of done. I always like to learn better ways.

---------------------------------------------------------------------------------
Could you run them through sort and use the -u option (unique)?

---------------------------------------------------------------------------------
Can you post an example of the source data?
Also I would recommend replacing all the hardcoded paths and filenames with variables and declaring them at the top (with meaningful names to improve readability and portability), e.g. change

ls /export/home/uprdjde1/scripts/JDE_* > tmp.txt
VAR1=tmp.txt

temp_outfile1=/tmp/${0}${$}_1.txt

scripts='/export/home/uprdjde1/scripts/JDE_*'
ls ${scripts} > ${temp_outfile1}

One common standard is to use uppercase only for exported variables, otherwise only lowercase (or is that just me?)

---------------------------------------------------------------------------------
Here is one of the source files look like, there is about 6 to 8 of them.
USER=$1
export USER
PASS=$2
export PASS
ENVIRON=$3
export ENVIRON
DBNAME=$4
export DBNAME

date +"%D %T %t J D Edwards Daily 17:00 Batch Stream"

date +"%D %T %t JDE $ENVIRON Step 1 - Post JDE Invoice Transactions"
/export/home/uprdjde1/scripts/runube.scr $USER $PASS $ENVIRON R09801 SFR0002

date +"%D %T %t JDE $ENVIRON Step 2 - Post JDE Cash Receipts Transactions"
/export/home/uprdjde1/scripts/runube.scr $USER $PASS $ENVIRON R09801 SFR0003

date +"%D %T %t JDE $ENVIRON Step 3 - Post JDE RR Transactions"
/export/home/uprdjde1/scripts/runube.scr $USER $PASS $ENVIRON R09801 SFR0005

date +"%D %T %t JDE $ENVIRON Step 4 - Run SQL Plus Script UMJDRRC1"
/export/home/uprdjde1/scripts/sqlplus_mail.scr umjdrrc1 $DBNAME $ENVIRON
if [ $? != 0 ]; then
exit 009
fi

date +"%D %T %t JDE $ENVIRON Step 5 - Run SQL Plus Script UMJDRRC2"
/export/home/uprdjde1/scripts/sqlplus_mail.scr umjdrrc2 $DBNAME $ENVIRON
if [ $? != 0 ]; then
exit 009
fi

The big problem I have is not all the title is the same 2nd column. So if there is away to only compare the first column and get ride of the dups that way.

Thanks for the help all and I will update my script with your suggestions.

---------------------------------------------------------------------------------
Is this more of what you were talking about?
#!/usr/bin/ksh

InputFile="/export/home/uprdjde1/scripts/JDE_*"
InputFile1="/export/home/uprdjde1/scripts"
temp_out0=/tmp/${0}${$}_0.txt
temp_out1=/tmp/${0}${$}_1.txt
temp_out2=/tmp/${0}${$}_2.txt

ls ${InputFile} > ${temp_out0}

while txtline=$(line)
do
echo "$txtline" |cut -d"/" -f6 | read DIR

egrep "Step|${InputFile1}" "$txtline" > ${temp_out1}

while txtline2=$(line)
do
echo "$txtline2" |cut -d" " -f1 | read INPUT
echo "$txtline2" |cut -d" " -f2 | read JOB1
echo "$txtline2" |cut -d" " -f5 | read JOB2
echo "$txtline2" |cut -d" " -f6 | read JOB3
echo "$txtline2" |cut -d" " -f3 | read JOB4
if [ "$INPUT" = "${InputFile1}/runube.scr" ]
then
echo "$JOB2" >> JDE_RunList.txt
elif [ "$INPUT" = "${InputFile1}/jde_rc.pl" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "${InputFile1}/sqlplus_mail.scr" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "${InputFile1}/sqlplus.scr" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "${InputFile1}/filewatch.pl" ]
then
JOB1=`echo "$JOB1" | tr "[/]" "[ ]"`
echo "$JOB1" |cut -d " " -f5 | read TRIGGER
echo "TriggerJob" >> JDE_RunList.txt
elif [ "$INPUT" = "${InputFile1}/sqlloader.scr" ]
then
echo "SQLLoader" >> JDE_RunList.txt
elif [ "$INPUT" = "${InputFile1}/jde_rc.pl" ]
then
echo "$JOB1" >> JDE_RunList.txt
elif [ "$INPUT" = "date" ]
then
echo "$txtline2" |cut -d"-" -f2 | read TITLE
echo "$TITLE:" >> JDE_RunList.txt
fi
done < ${temp_out1}
done < ${temp_out0}

awk '{printf;getline;print " " $0}' JDE_RunList.txt > ${temp_out2}

while txtline3=$(line)
do
echo "$txtline3" |cut -d":" -f1 | read SWITCH1
echo "$txtline3" |cut -d":" -f2 | read SWITCH2
printf '%-12s%-48s%-9s%-10s%-19s\n' "$SWITCH2" "$SWITCH1" "c.JDE " "cProcess " "device.cProcess" >> JDE_JobList.txt
done < ${temp_out2}
sort JDE_JobList.txt > JDE_JobList1.txt
cat JDE_JobList1.txt | tr "[\"]" "[ ]" > JDE_JobList.txt

\rm JDE_RunList.txt
\rm JDE_JobList1.txt
\rm ${temp_out0}
\rm ${temp_out1}
\rm ${temp_out2}

---------------------------------------------------------------------------------
Hello all I found the command I was looking for here is what I used
sort -uk 1,1 JDE_JobList1.txt > JDE_JobList.txt
Thanks for all the help William if you can find other improvement to my script please let me know I'm always tring to improve my script writing.
Thanks all for the help

---------------------------------------------------------------------------------
Not sure I was much help, but script is easier to follow now anyway.
I hadn't come across the $(line) function - is that Solaris? I would have used something like

while read input job1 job4 filler job2 job3 others
do
print input = $input, job1 = $job1, job2 = $job2, job3 = $job3, job4 = $job4
done < filename.ext

which reads fields directly into variables avoiding all the 'echo' and 'cut' lines.

Also I prefer to use Korn shell [[ ]] test expressions (double square brackets) rather than the old-fashioned Bourne [ ] ones as I find them more reliable at comparing wildcard expressions, for example

$ test=bananawilliamhatstand
$ [[ $test = *william* ]] && print Yep || print Nope

Yep

$ [ $test = *william* ] && print Yep || print Nope

Nope

---------------------------------------------------------------------------------
Thanks I will work on updating the script. Thanks for the info I will try and use it on my next one.

Quick Links:
Do you have a UNIX Question?

Unix Home: Unix System Administration Hints and Tips