Products
Contact Support Additional Learning Monthly Product Updates
Menu

Forsta Surveys Support

All the topics, resources needed for Forsta Surveys.

Advanced Unix Commands for Programmers

  • Last updated:

Articles in this section

See more
In this article

     Requires Decipher Cloud

    Some things to note before you begin:

    • Almost all Unix commands are case sensitive
    • After almost every command, you can add --help to find out more information (e.g. grep --help)
    • The asterisk (*) is used to match any and all characters
      (e.g. */survey.xml matches the survey.xml file in every directory from your current position)
    • The period . means the current directory
    • The double period .. is the previous directory
    • Chain commands together with the pipe | symbol (e.g., ls | grep "proj14" | sort)
    • Output the results of any command with the greater-than symbol >
      (e.g. ls > directory_contents.txt # creates/overwrites the directory_contents.txt file)
      (e.g. ls >> directory_contents.txt # appends to the directory_contents.txt file)
    • Use CTRL + C to cancel a current running command. (Press repeatedly if needed)

    1:  Grep (i.e., Searching for a String in a File)

    The unix command `grep` is a powerful tool that can be used to search for strings within files.

    For example, if we wanted to find out what kind of question "Q1" is in our survey.xml file, we could run the following command from our project's directory:

    Input

    grep 'label="Q1"' survey.xml

    Output

    <radio label="Q1">

    Since we found an exact match in the survey.xml file, we can see that Q1 is a <radio> question.

    What if we wanted to see what type of question "Q1" is for every project in our company directory? We can use the asterisk to match all directories containing a survey.xml from our company's directory:

    Input

    grep 'label="Q1"' */survey.xml

    Output

    proj13040b/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13040/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13041/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13042/survey.xml:<radio label="Q1" shuffle="rows" colLegendRows="9">
proj13043/survey.xml:<textarea label="Q1" optional="0">
proj13050/survey.xml:<text label="Q1" optional="0" style="exclusiveCol">
proj13052/survey.xml:<radio label="Q1" optional="0" randomize="0">
proj13055/survey.xml:<checkbox label="Q1" atleast="1">
proj13056/survey.xml:<checkbox label="Q1" atleast="1">
proj13059/survey.xml:<select label="Q1" optional="0">

    We can see that "Q1" reflects a variety of question types in several of our projects.

    In that same example, if we only wanted to view the directories where "Q1" was used as a label, we can add -l to our command to print only names of files that contain a match.

    Input

    grep -l 'label="Q1"' */survey.xml

    Output

    proj13040b/survey.xml
proj13040/survey.xml
proj13041/survey.xml
proj13042/survey.xml
proj13043/survey.xml
proj13050/survey.xml
proj13052/survey.xml
proj13055/survey.xml
proj13056/survey.xml
proj13059/survey.xml

    Only the files that contain an exact match were returned. Perfect.

    To perform a case-insensitive search, in case "Q1" was written as "q1", we can add -i to our command:

    Input

    grep -i 'label="Q1"' */survey.xml

    Output

    proj120200/survey.xml:<radio id="sJtiq" optional="0" label="q1" builder:autoGrouping="1">
proj120701/survey.xml:<number id="UZLDT" label="q1" optional="0" size="10">
proj120800/survey.xml:<number id="UZLDT" label="q1" optional="0" size="10">
proj13040b/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13040/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13041/survey.xml:<checkbox label="Q1" atleast="1" shuffle="rows">
proj13042/survey.xml:<radio label="Q1" shuffle="rows" colLegendRows="9">
proj13043/survey.xml:<textarea label="Q1" optional="0">
proj13050/survey.xml:<text label="Q1" optional="0" style="exclusiveCol">
proj13052/survey.xml:<radio label="Q1" optional="0" randomize="0">
proj13055/survey.xml:<checkbox label="Q1" atleast="1">
proj13056/survey.xml:<checkbox label="Q1" atleast="1">
proj13059/survey.xml:<select label="Q1" optional="0">

    With the results above, we can see that projects with a lable of "q1" and "Q1" were now shown.

    We can use -A# and -B# to also grab a number of lines after or before the line that matches our search. The command below will search for all radio questions with a label of "Q1" in each survey.xml for every directory in our current directory.

    Input

    grep -A2 '<radio label="Q1"' */survey.xml

    Output

    proj120500/survey.xml: <radio label="Q1" shuffle="rows" id="DUAP7">
proj120500/survey.xml-    <title id="sg7J9">Overall how familiar are you with the 4:30 to 7:00 a.m. local newscasts on the following news stations?</title>
proj120500/survey.xml-    <row label="r1" id="yPuQF">Channel 3 - KSNV</row>
--
proj130500/survey.xml: <radio label="Q1" id="I0NS5">
proj130500/survey.xml-    <title id="AfWT7">How many hours a week do you spend playing computer or video games?</title>
proj130500/survey.xml-    <row label="r1" id="fHsBI">Less than 5 hours per week</row>
--
proj13042/survey.xml:<radio label="Q1" shuffle="rows" colLegendRows="9">
proj13042/survey.xml-  <title>
proj13042/survey.xml-    How often do you do the following?
--
proj13052/survey.xml:<radio label="Q1" optional="0" randomize="0">
proj13052/survey.xml-   <title>Do you <u>watch</u>, or record to view at a later time, any daytime dramas or soap operas that air on television, including cable or satellite?</title>
proj13052/survey.xml-   <comment>Select one<br/><br/></comment>--

    The command grabbed the line containing our match plus 2 more lines following the matched line.

    If we had the following email list:

    horse@coconuts.com
knight@ni.com
swallow[at]unknownlocation.com
sirlancelot@camelot.com
troll@bridge.com

    To find the lines that don't contain a valid email address with the "@" symbol, we can add -v to our grep command to select non-matching lines.

    Input

    grep -v "@" email-list.txt

    Output

    swallow[at]unknownlocation.com

    Learn more:

    man grep

    2:  Find (i.e., Searching for a File in a Directory)

    The unix command `find` will navigate child directories for whatever string you want to find:

    To get a feel for what find does, type it in and see what happens.

    Input

    find

    For example, if you want to search for all directories containing a /lang directory, use the following command:

    Input

    find -name lang

    Output

    ./proj14005/temp-final-20140109/lang
./proj14005/lang
./proj14005/temp-spanish/lang
./proj13104/lang
./proj13118a/lang
./proj13108/temp-final-tes/lang
./proj13108/temp-09-04/lang
./proj13108/temp-final-tes/lang
./proj13108/lang

    Use the find command and pipe the results to grep to achieve a similar effect.

    Input

    find | grep lang

    Output

    ./proj14005/temp-final-20140109/lang
./proj14005/lang
./proj14005/temp-spanish/lang
./proj13104/lang
./proj13118a/lang
./proj13108/temp-final-tes/lang
./proj13108/temp-09-04/lang
./proj13108/temp-final-tes/lang
./proj13108/lang

    Similarly, you can add another pipe to sort the results.

    Input

    find | grep lang | sort

    Output

    ./proj13104/lang
./proj13108/lang
./proj13108/temp-09-04/lang
./proj13108/temp-final-tes/lang
./proj13108/temp-final-tes/lang
./proj13118a/lang
./proj14005/lang
./proj14005/temp-final-20140109/lang
./proj14005/temp-spanish/lang

    The results have been sorted alphabetically in ascending order.

    Now let's find all lang directories that were created in 2014.

    Input

    find proj14*/ -name lang

    Output

    proj14005/temp-final-20140109/lang
proj14005/lang
proj14005/temp-spanish/lang

    3:  Head & Tail (i.e., Reading a Number of Lines)

    Head - output the first part of files

    Tail - output the last part of files

    To get a feel for what these commands do, choose a file (e.g. file.txt) and running the following commands:

    head -2 file.txt
tail -5 file.txt

    The head command above will output the first two lines of the file and the tail command will output the last 5 lines of the file. The numbers provided can be any number.

    For example, if you have a list of emails (e.g. email-list.txt) to send your campaign to, you could get a soft send list (e.g. soft-send.txt) of 5000 using the following command:

    head -5000 email-list.txt > soft-send.txt

    This will create/overwrite the soft-send.txt file with the first 5000 lines of our email-list.txt file.

    To create a full-send.txt file containing the remaining email addresses to send to, we would then find out how many lines exist in the email-list.txt and use the tail command to get the rest of the list. Assuming there were 50,000 lines, the following command would do:

    tail -45000 email-list.txt > full-send.txt

    Similarly, a bit more advanced approach would be to calculate the number of lines using multiple commands.

    tail -$(expr $(cat email-list.txt | wc -l) - 5000) email-list.txt > full-send.txt

    The example above uses 3 different sets of commands to achieve the final result.

    cat list.txt | head -n 10001 > list.softSend.txt

    4:  Sort

    The `sort` command will sort content for you. Look at the following workflow to get a feel for what sort is capable of.

    $ cat list.txt
4
a
3
d
2
a
b
4
7
1
$ sort list.txt
1
2
3
4
4
7
a
a
b
d
$ sort -r list.txt
d
b
a
a
7
4
4
3
2
1
$ cat list.txt | sort | uniq
1
2
3
4
7
a
b
d

    Learn more:

    man sort

    5:  awk

    awk is an advanced tool used to process and alter files, one line at a time.

    To begin, let's take the following tab-delimited file (list.txt) and process it using awk.

    1 2 3 4 5
1 2 3 4 5
1 2 3 p 5
1 2 3 4 5

    To iterate through each line and print the entire line:

    Input

    awk '{print $0}' list.xt

    Output

    1 2 3 4 5
1 2 3 4 5
1 2 3 p 5
1 2 3 4 5

    Nothing magical here, just note that $0 means the entire line. If we only want to print the 3rd column:

    Input

    awk '{print $3}' list.xt

    Output

    3
3
3
3

    If we wanted to change the 4th column to the number 6:

    Input

    awk '{$4 = "6"; print $0}' list.txt

    Output

    1 2 3 6 5
1 2 3 6 5
1 2 3 6 5
1 2 3 6 5

    If we wanted to change the 4th column to the number 6 only if the 4th column is the number 4:

    Input

    awk '{if ($4 == "4") $4 = "6"; print $0}' list.txt

    Output

    1 2 3 6 5
1 2 3 6 5
1 2 3 p 5
1 2 3 6 5

    Learn more:

    man awk

    6:  Find and Replace with tr

    The `tr` command is able to replace or remove characters inputted into it. The number of characters must be the same.

    In this example, we replace all spaces with tabs:

    cat file.txt | tr ' ' '\t' > out.txt

    As shown below, we can also use tr with sets.

    $ echo "testing" | tr "a-z" "A-Z"
TESTING
$ echo "tesTing" | tr "[:lower:]" "[:upper:]"
TESTING
$ echo "testing123" | tr "[:alpha:]" "a"
aaaaaaa123

    7:  Find & Replace with sed

    The command `sed` uses regular expressions to find and replace content within a file. For example, to replace all occurrences of "foo" with "foobar" in file.txt, use the following command:

    cat file.txt | sed "s/foo/foobar/g" > out.txt

    Note: The "/g" means find and replace to every instance in the file.

    You can use sed to quickly add a new column to a file:

    sed "s/^/foo\t/g" input.txt > out.txt

    You can use sed to add one to the end of the file:

    sed "s/$/\tfoo/g" input.txt > output.txt

    To find all the IF's or exec's within a survey.xml:

    Input

    cat survey.xml |  sed -n '/<if cond/,/<\/if>/p' > out.txt or cat survey.xml |  sed -n '/<exec>/,/<\/exec>/p' > out.txt

    Output

    <if cond="'nolist' in p.markers or 'noclist' in p.markers">
  <comment label="missing" where="survey"><div class='error'>
You are missing information in the URL. Please verify the URL with the original invite.
</div>;
<br/><br/></comment>

  <suspend/>

  <goto target="missing"/>
</if>
<if cond="(1)" randomize="1">
<exec>
_index = ord("$(_letter)") - ord("a") + 1

p._item = "i%s" % _index
p._design = p._designs[0]
p._seller = p._sellers[_index - 1]

p._itemOrder.append(p._item)
</exec>

    8:  seq (i.e., Generate a Sequence of Numbers)

    The `seq` command can be used to quickly generate a series of numbers. For example, to output the numbers 1 - 100 to a file:

     seq 1 100 > numbers.txt 

    1
2
3
4
5
6
7
8
9
10
...
99
100

    Similarly, we can print out every odd from 1 - 100:

     seq 1 2 100 > numbers.txt 

    1
3
5
7
9
...
99

    Learn more:

    seq --help

    9:  wc (i.e., Word Count)

    The `wc` command can be used to count lines, characters, or letters from a file. Given the following file (file.txt):

    alpha
omega
beta

    We can count the number of lines:

    Input

    cat file.txt | wc -l

    Output

    3

    We can count the number of characters:

    Input

    cat file.txt | wc -c

    Output

    17

    We can count the number of words:

    Input

    cat file.txt | wc -w

    Output

    3

    Learn more:

    wc --help

    10:  --help & man

    To learn more about any Unix command, you can append --help after the command for more information:

    grep --help
wc --help
seq --help
cat --help
ls --help

    If --help is not available, you can view the man page for any command with the man command (stands for manual):

    man grep
man wc
man seq
man cat
man ls

    Note:  Scripts that were created solely for the Decipher platform may not contain a man page.

    11:  Randomly generated list file

    Need a list file with random data in it? Run:

    here dev/generate-random -n500 source email name list:1,2,3 country:uk,us,eu > somefile.txt

    The -n500 determines count of items in the list. The fields source, email and name have a random appropriate value. For other fields, specify the field name, then add a colon and a comma-separated list of values.

    12:  View Email Send Statistics

    You can use the analyze-email script to view the email send statistics for any company in the Portal. This script aggregates information from email campaigns, optionally splitting the results by a variable.

    The following data is returned per list:

    • Project path
    • List name (displayed only for initial sends)
    • Number of emails sent
    • Number of click-throughs
    • Number of qualified completes
    • Send date

    Either specific surveys or all surveys within a company can be analyzed. For example, the following command reads all surveys assigned to the specified company:

    here dev/analyze-email –split co –company "[COMPANY NAME]" > file.txt

    The following command reads only the specified survey(s) assigned to the specified company:

    here dev/analyze-email –split co [SURVEY PATH 1] [SURVEY PATH 2] > file.txt

    Note: The -split option is not mandatory; if it is not set, then no split is applied.

    Was this article helpful?
    0 out of 0 found this helpful
    Return to top