11. Data Analytics - Prepare Data for Exploration - Week 2

Definition:

Bias // a preference in favor of or against a person, group of people, or thing

Data bias // a type of error that systematically skews results in a certain direction

Sampling bias // when a sample isn't representative of the population as a whole

Unbiased sampling // when a sample is representative of the population being measured

Ethics // well-founded standards of right and wrong that prescribe what humans ought to do, usually in terms of rights, obligations, benefits to society, fairness, or specific virtues

Data ethics // well-founded standards of right and wrong that dictate how data is collected, shared, and used

GDPR // general data protection regulation of the European union

Data anonymization // process of protecting people's private or sensitive data by eliminating personally identifiable information (PII). Processes: blanking, hashing, masking, or using codes or altered texts.

De-identification // process used to wipe data clean of all personally identifying information

Data interoperability // ability of data systems and services to openly connect and share data. is used in healthcare industry.


What data should be anonymized?

    - healthcare and financial data

    - numbers, names, license plates/numbers, SSN, IP addresses, medical records, emails, photos, etc


Aspects of data ethics

    - ownership:

            Individuals own the raw data they provide and they have primary control over its usage,

            how it's processed, and how it's shared.

    - transaction transparency

            All data-processing activities and algorithms should be completely explainable and understood

            by the individual who provides their data.

    - consent

            an individual right to know explicit details about how and why their data will be used

            before agreeing to provide it.

    - currency

            Individuals should be aware of financial transactions resulting from the use of their personal

            data and the scale of these transactions.

    - privacy

            preserving a data subject's information and activity any time a data transaction occurs.

                - Protection from unauthorized access to our private data.

                - Freedom from inappropriate use of our data.

                - The right to inspect, update, or correct our data.

                - Ability to give consent to use our data.

                - Legal right to access the data.

    - openness (or open data)

            Free access, usage, and sharing of data

                - Availability and access // available and accessable

                - Reuse and redistribution // allows reuse and redistribution of data

                - Universal participation // no restrictions on who can use the data


Identifying good data:

    R = reliable. not biased

    O = original. first party data.

    C = comprehensive. contains all information needed to answer question or solution

    C = current. usefulness of data decreases as time passes

    C = cited. makes information credible

    *good data rocccs!

Identifying bad data:

    opposite of ROCCC.

    Every good solution is found by avoiding bad data.

    

Types of data bias:

    - sampling bias 

            When a sample isn't representative of the population as a whole

    - observer bias (experimenter bias/research bias)

            The tendency for different people to observe things differently 

    - interpretation bias

            The tendency to always interpret ambiguous situations in a positive or negative way

    - confirmation bias

            The tendency to search for or interpret information in a way that confirms pre-existing beliefs


Bias Occurs:

    - during data collection

    - during planning. not being inclusive

    - subconsciously or consciously

Solution:

    - choose data randomly in a population


Ensuring Data Integrity Process:

    - Analyze data for bias and credibility

    - Good vs. bad data

    - Data ethics, privacy, and access


Additional Resources:

https://www.data.gov/

https://www.census.gov/data.html

https://www.opendatanetwork.com/

https://cloud.google.com/public-datasets

https://datasetsearch.research.google.com/

Comments

Post a Comment

Popular posts from this blog

2. FreeCodeCamp - Dynamic Programming - Learn to Solve Algorithmic Problems & Coding Challenges

Dynamic Programming - Parts      PART 1 - Memoization     P ART 2 - Tabulation Dynamic Programming with Fibonacci Example Recursion: Slow because time complexity is O(2^n) func fibR (x int ) int {     if x <= 2 {         return 1     }     return fibR (x- 1 ) + fibR (x- 2 ) } Dynamic Way Example Recursion with Stored Value:  // Fast because it removes duplicate processes by checking map. // Tail recursion-ish // Results are hitchhiking each recursive call func fibR2 (x int , y map [ int ] int ) int {     if _ , ok := y[x]; ok {         return y[x]     }     if x <= 2 {         return 1     }     y[x] = fibR2 (x- 1 , y) + fibR2 (x- 2 , y)     return y[x] }
Read more

20. Data Analytics - Analyze Data to Answer Questions - Week 1

Goal of analysis is to identify trends and relationships within data so you can accurately answer the questions you're asking. Most data is organized in tables, always have data in the right format. Definition : Analysis // process used to make sense of the data collected Sorting // involves arranging data into a meaningful order to make it easier to understand, analyze, visualize Outliers // data points that are very different from similarly collected data and might not be reliable values Filtering // extracting specific data that meet certain criteria while hiding the rest Customized sort order // when you sort data in a spreadsheet using multiple conditions The 4 phases of analysis:          1. Organize data // getting the right data in the right size and order                     - Sorting : arrange data in meaningful order                     - Filtering : can help find errors or outliers          2. Format and adjust data // filtering or sorting or compiling data in a better
Read more

3. Algorithms - Selection Sort

Selection sort sorts an array by finding the lowest value element and appending it to an empty array. The algorithm will shrink the unsorted array and repeat the process recursively until all unsorted elements are used up. package main func selectionSort (list *[] int ) {     sorted := make ([] int , 0 , len (*list))     unSorted := *list     for i := 0 ; i < len (*list); i++ {         min := unSorted[ 0 ]         index := 0         for i , e := range unSorted {             if e <= min {                 min = e                 index = i             }         }         sorted = append (sorted, (unSorted)[index])         if index == 0 {             unSorted = unSorted[ 1 :]         } else if index == ( len (*list) - 1 ) {             unSorted = unSorted[:index]         } else {             unSorted = append (unSorted[ 0 :index], unSorted[index+ 1 :]...)         }     }     * list = sorted } /*func main() {     x := []int{4, 6, 3, 2, 9, 7, 3, 5}     fmt.Print
Read more