TADA Package Training: A Markdown For Region 5 R Users Network
TADA Team
2024-03-29
Source:vignettes/TADAR5.Rmd
TADAR5.RmdWelcome
Thank you for your interest in Tools for Automated Data Analysis (TADA). TADA is an open-source tool set built in the R programming language. This RMarkdown document walks users through how to download the TADA R package from GitHub, access and parameterize several important functions, and create basic visualizations with a sample data set. The sample data set contains data from one week from all EPA Region 5 states.
Note: TADA is still under development. New functionality is added weekly, and sometimes we need to make bug fixes in response to tester and user feedback. We appreciate your feedback, patience, and interest in these helpful tools.
If you are interested in contributing to TADA development, more information is available at [Contributing] (https://usepa.github.io/TADA/articles/CONTRIBUTING.html). We welcome collaboration with external partners.
Install and load packages
First, install and load the remotes package specifying the repo. This is needed before installing TADA because it is only available on GitHub.
install.packages("remotes",
repos = "http://cran.us.r-project.org"
)
library(remotes)Next, install and load TADA using the remotes package. TADA R Package dependencies will also be downloaded automatically from CRAN with the TADA install. You may be prompted in the console to update dependency packages that have more recent versions available. If you see this prompt, it is recommended to update all of them (enter 1 into the console).
remotes::install_github("USEPA/TADA",
ref = "develop",
dependencies = TRUE
)Finally, use the library() function to load the TADA R Package into your R session.
Help pages
All TADA R package functions have their own individual help pages,
listed on the Function
reference page on the GitHub site. Users can also access the help
page for a given function in R or RStudio using the following format
(example below): ?TADA::[name of TADA function]
# Access help page for TADA_DataRetrieval
?TADA_DataRetrievalRetrieve WQP data
WQP data is retrieved and processed for compatibility with TADA. This function, TADA_DataRetrieval, builds on USGS’s dataRetrieval R package functions. It joins three WQP profiles: Site, Sample Results (physical/chemical metadata), and Project. In addition, it changes all data in the Characteristic, Speciation, Fraction, and Unit fields to uppercase, removes exact duplicates, and addresses result values that include special characters.
The characteristics pulled from Water Quality Portal (WQP) are not restricted and can include:
startDate
endDate
characteristicName
sampleMedia
siteType
statecode (review list of possible state and territory abbreviations)
countycode
siteid
organization
project
huc
characteristicType
The default TADA_DataRetrieval function automatically runs the TADA_AutoClean function. In this example, we will set TADA_AutoClean = FALSE and run it as a separate step in the work flow.
Tips:
-
All the query filters for the WQP work as an AND but within the fields there are ORs. For example:
Characteristics: If you choose pH & DO - it’s an OR. You will retrieve both pH OR DO data if available.
States: Similarly, if you choose MI and IL, it’s an OR. You will retrieve both MI OR IL data if available.
Combinations of fields are ANDs, such as State/MI AND Characteristic/DO. This means you will receive all DO data available in MI.
Characteristic and Characteristic Type also work as an AND. This means that the Characteristic must fall within the CharacteristicGroup if both filters are being used, if not you will get an error.
The “siteid” is a general term WQP uses to describe both Site IDs from USGS databases and Monitoring Location Identifiers (from WQX). Each monitoring location in the Water Quality Portal (WQP) has a unique Monitoring Location Identifier, regardless of the database from which it derives.
The example dataset in this demo includes monitoring data from multiple organizations collected during one week (May 1, 2019 - May, 7 2019) in Region 5 states (Illinois, Indiana, Michigan, Minnesota, Ohio, and Wisconsin).
For demonstration purposes, the R5 example data set has already been downloaded. You can run the code chunk below to access the pre-downloaded data set. The original query has also been included and can be accessed by uncommenting and running the TADA_DataRetrieval function below.
Downloads using TADA_DataRetrieval will have the same columns each time. Any columns with a “TADA” prefix were added during data retrieval. TADA_DataRetrieval automatically converts the date times to UTC and data to dates, datetimes, and numerics based on a standard algorithm. Data are uploaded to the Water Quality Portal by individual organizations, which may or may not follow the same conventions. Data and metadata quality are not guaranteed! Carefully explore any data and make conservative quality assurance decisions where information is limited.
Note: USGS and EPA are working together to create WQP 3.0 data profiles. Once released, one data profile will contain the columns critical to TADA, removing the need to combine profiles. TADA package users likely will not notice a difference in their usage of the TADA_DataRetrieval function, but it will simplify the steps needed to upload a custom or WQP GUI-downloaded data set into the R package.
# Uncomment query below to download data set from WQP
# TADAProfile <- TADA_DataRetrieval(statecode = c("IL", "IN", "MI", "MN", "OH", "WI"), startDate = "2019-05-01", endDate = "2019-05-07", applyautoclean = FALSE)
# For demo purposes, import pre-downloaded R5 data set
R5Profile <- TADA::Data_R5_TADAPackageDemoR5 Profile
The table above shows the first 100 records from the R5 Water Quality Portal Query.
Question 1: How many results did TADA_DataRetrieval return?
# Determine number of rows in the data set
R5_nresults <- nrow(R5Profile)Answer: There are 171988 results in the data set.
If you need to download a large amount of data from across a large area, and the TADA_DataRetrieval function is not working due to WQP timeout issues, then the TADA_BigDataRetrieval function may work better. See ?TADA_BigDataRetrieval for more details or the TADA Module 1 vignette for more details.
Review and Filter By Media Type
TADA is primarily designed to accommodate water data from the WQP. Let’s see what activity media types are represented in the data set.
To help answer this question, we can use TADA functions to create a table or pie chart of the counts of unique values in a user-specified field with TADA_FieldValuesTable or TADA_FieldValuesPie. We’ll start with TADA.ActivityMediaName.
# Create pie chart for ActivityMediaName
ActMedName_Pie <- TADA_FieldValuesPie(R5Profile, field = "ActivityMediaName")
ActMedName_Pie
# Create table with count for each ActivityMediaName
FieldValues_ActMedia <- TADA_FieldValuesTable(R5Profile, field = "ActivityMediaName")Question 2: How many unique ‘ActivityMediaName’ values exist in your dataset? Are there any media types that are not water?
Answer: There are 6 unique ActivityMediaNames in the data set. They are: Water, Biological, Air, Sediment, Habitat and Other. 5 of these are not water.
Filter for Surface Water Data
Some TADA users are interested in using WQP data for surface water only or including some non-water data. The TADA_AnalysisDataFilter identifies surface water, groundwater, and sediment results. Multiple columns are used to identify groundwater results as organizations may populate different combinations of fields in order to identify groundwater results.
Users can specify whether all results should be returned with a new column, TADA.UseForAnalysis.Flag, providing information about the result’s media type and identifying if the result should be included in further analysis, or if only results that should be included are returned.
The defaults are to include surface water, exclude groundwater and sediment, and to return all results with the added TADA.UseForAnalysis.Flag column (clean = FALSE). This is shown in the example below.
We can use TADA_FieldValuesTable and TADA_FieldValuesPie again to review this more specific breakdown of results by media type.
Note: When WQX 3.0 data profiles are released, this function will be updated to include fish tissue a filter option.
# Filter to flag only surface water results for use in analysis
R5Profile <- TADA_AnalysisDataFilter(R5Profile, clean = FALSE)
#> [1] "TADA_AnalysisDataFilter: Identifying groundwater results."
#> [1] "TADA_AnalysisDataFilter: Flagging surface water results to include in assessments."
#> [1] "TADA_AnalysisDataFilter: Flagging groundwater results to exclude from assessments."
#> [1] "TADA_AnalysisDataFilter: Flagging sediment results to exclude from assessments."
#> [1] "TADA_AnalysisDataFilter: Returning all results with TADA.UseForAnalysis.Flag column indicating if result should be used for assessments."
# Create pie chart for TADA.UseForAnalysis.Flag
FieldVal_Pie <- TADA_FieldValuesPie(R5Profile, field = "TADA.UseForAnalysis.Flag")
FieldVal_Pie
FieldValues_AnalysisFlag <- TADA_FieldValuesTable(R5Profile, field = "TADA.UseForAnalysis.Flag")Question 3: How many SURFACE WATER results can we retain for further analysis
Answer: There are 166127 surface water results in the R5 data set. Let’s filter out any other results flagged “No” to reduce the size of the data set before continuing.
AutoClean
Now TADA_AutoClean can be run on the surface water-only data set. It performs the following functions on the data retrieved from the WQP:
TADA_ConvertSpecialChars - converts result value columns to numeric and flags non-numeric values that could not be converted.
TADA_ConvertResultUnits - unifies result units for easier quality control and review
TADA_ConvertDepthUnits - converts depth units to a consistent unit (meters). This function can also be run on its own, for example if you would like to convert depths from meters to feet later in the workflow.
TADA_IDCensoredData - categorizes detection limit data and identifies mismatches in result detection condition and result detection limit type.
Other helpful actions - converts important text columns to all upper-case letters, removes exact duplicates, and uses WQX format rules to harmonize specific NWIS metadata conventions (e.g. move characteristic speciation from the TADA.ResultMeasure.MeasureUnitCode column to the TADA.MethodSpeciationName column)
As a general rule, TADA functions do not change any contents in the WQP-served columns. Instead, they add new columns with the prefix “TADA.” The following columns are numeric versions of their WQP origins: TADA.ResultMeasureValue, TADA.DetectionQuantitationLimitMeasure.MeasureValue, TADA.LatitudeMeasure, and TADA.LongitudeMeasure.
These functions also add the columns TADA.ResultMeasureValueDataTypes.Flag and TADA.DetectionQuantitationLimitMeasure.MeasureValueDataTypes.Flag, which provide information about the result values that is needed to address censored data later on. These new columns flag if special characters are included in result values, and specify what the special characters are.
# run TADA_AutoClean on filtered dataset to convert special characters, result units, and depth units and identify censored data.
R5Profile <- TADA_AutoClean(R5Profile)
#> [1] "TADA_Autoclean: creating TADA-specific columns."
#> [1] "TADA_Autoclean: harmonizing dissolved oxygen characterisic name to DISSOLVED OXYGEN SATURATION if unit is % or % SATURATN."
#> [1] "TADA_Autoclean: handling special characters and coverting TADA.ResultMeasureValue and TADA.DetectionQuantitationLimitMeasure.MeasureValue value fields to numeric."
#> [1] "TADA_Autoclean: converting TADA.LatitudeMeasure and TADA.LongitudeMeasure fields to numeric."
#> [1] "TADA_Autoclean: harmonizing synonymous unit names (m and meters) to m."
#> [1] "TADA_Autoclean: updating deprecated (i.e. retired) characteristic names."
#> [1] "518 results in your dataset have one of the following deprecated characteristic names: 1,2-Benzenedicarboxamide, N2-[1,1-dimethyl-2-(methylsulfonyl)ethyl]-3-iodo-N1-[2-methyl-4-[1,2,2,2-tetrafluoro-1-(trifluoromethyl)ethyl]phenyl]-; Desisopropyl atrazine; Inorganic nitrogen (nitrate and nitrite); Inorganic nitrogen (nitrate and nitrite) ***retired***use Nitrate + Nitrite; Nutrient-nitrogen***retired***use TOTAL NITROGEN, MIXED FORMS with speciation AS N. These names have been substituted with the updated preferred names in the TADA.CharacteristicName field."
#> [1] "TADA_Autoclean: harmonizing result and depth units."
#> [1] "TADA_Autoclean: creating TADA.ComparableDataIdentifier field for use when generating visualizations and analyses."
#> [1] "NOTE: This version of the TADA package is designed to work with numeric data with media name: 'WATER'. TADA_AutoClean does not currently remove (filter) data with non-water media types. If desired, the user must make this specification on their own outside of package functions. Example: dplyr::filter(.data, TADA.ActivityMediaName == 'WATER')"Monitoring Location Review
Now let’s take a look at a breakdown of these monitoring location types. The previous function removed non-surface water results. Depending upon your program’s goals and methods, you might want to further filter the monitoring location types in the data set..
MonLocTypNam_Pie <- TADA_FieldValuesPie(R5Profile, field = "MonitoringLocationTypeName")
MonLocTypNam_Pie
Question 4: How many unique Monitoring Location Types (MonitoringLocationTypeName) are present? Which is the most common?
FieldValues_MLs_table <- TADA_FieldValuesTable(R5Profile, field = "MonitoringLocationTypeName")
mlt_n <- length(unique(FieldValues_MLs_table$Value))
mlt_most_common <- FieldValues_MLs_table %>%
dplyr::slice_max(Count)Answer: There are 19 unique MonitoringLocationTypeNames the R5 data set. The most common is River/Stream (n = 143613).
Now, let’s filter by River/Stream and focus the rest of this demo on the River/Stream subset of results.
R5Profile <- R5Profile %>%
dplyr::filter(MonitoringLocationTypeName == dplyr::pull(mlt_most_common[1]))Next we can review station locations and summary information using the TADA_OverviewMap function. TADA_OverviewMap counts the number of unique results, characteristics, and organizations at each monitoring location and creates a map for reviewing summary stats. Larger point sizes indicate more results, while darker blue colors indicate more unique characteristics collected at the site. Users may click on a site to view a pop-up with this summary information. This can be a good first step in identifying incorrect coordinates that are outside of the desired state(s) of interest or outside of the US.
OvwMap <- TADA_OverviewMap(R5Profile)
OvwMapAnother function, TADA_FlagCoordinates, identifies and flags potentially invalid coordinate data. Its functionality will not be showcased in this demo due to time constraints. For details see TADA: Module 1 or enter ?TADA_FlagCoordinates in the R console for more information and examples.
Summarize by Characteristic
TADA_SummarizeColumn summarizes the data set based on a user specified column and returns a data frame displaying the number of sites and number of records for each unique value in the specified column. This example uses TADA.CharacteristicName.
# Review the number of sites and number of records for each CharacteristicName in TADAProfile
R5Profile_CharSummary <- TADA_SummarizeColumn(R5Profile, "TADA.CharacteristicName")Question 5: How many unique values of TADA.CharacteristicName are included in the data set? Which TADA.CharactersticName was collected at the greatest number of sites? Which TADA.CharactersticName has the most results?
# Number of unique values of TADA.CharacteristicName
n_char_name <- length(unique(R5Profile_CharSummary$TADA.CharacteristicName))
# Filter for record with max number of sites per Character
max_sites <- R5Profile_CharSummary %>%
dplyr::slice_max(n_sites)
# Filter for record with max number of results per Character
max_results <- R5Profile_CharSummary %>%
dplyr::slice_max(n_records)Answer: There are 299 unique values of TADA.CharacteristicName in the R5 data set. TEMPERATURE, WATER was collected at the greatest number of sites (n = 547). TEMPERATURE, WATER has the most results (n = 547).
Statistically Aggregated Data
TADA_FindContinuousData checks for and removes statistically aggregated high frequency (i.e., continuous) data, if present.
The Water Quality Portal (WQP) is not currently designed to store high-frequency sensor data (more than 1 value per day). However, sometimes data providers choose to aggregate their continuous data to avg, max, or min value over daily or other intervals, and then submit that aggregated data to the WQP through WQX. This type of high frequency data may (or may not) be usable for your analyses. Therefore, this function uses metadata submitted by data providers to flag rows with aggregated continuous data. This is done by flagging results where the ResultDetectionConditionText = “Reported in Raw Data (attached)”.
When clean = TRUE, rows with aggregated continuous data are removed from the dataframe and no column will be appended. The default is clean = TRUE.
In this example, we’ll use the default clean = TRUE to remove any aggregated continuous data.
An additional input called flaggedonly will allow the user to filter data to show only rows of aggregated continuous data.
R5ProfileClean1 <- TADA_FindContinuousData(R5Profile)
#> [1] "No evidence of aggregated continuous data in your dataframe. Returning the input dataframe with TADA.AggregatedContinuousData.Flag column for tracking."WQX Quality Assurance and Quality Control (QAQC) Service Result Flags
Next we’ll run several functions to address invalid method, fraction, speciation, and unit metadata by characteristic. The default is clean = TRUE, which will remove invalid results. You can change this to clean = FALSE to flag results, but not remove them.
See documentation for more details:
-
?TADA_FlagMethod
When clean = FALSE, this function adds the following column to your dataframe: TADA.AnalyticalMethod.Flag. This column flags invalid TADA.CharacteristicName, ResultAnalyticalMethod/MethodIdentifier, and ResultAnalyticalMethod/MethodIdentifierContext combinations in your dataframe either “NonStandardized”, “Invalid”, or “Valid”.
When clean = TRUE, “Invalid” rows are removed from the dataframe and no column will be appended.
When flaggedonly = TRUE, the dataframe is filtered to only the rows flagged as “Invalid”; default is flaggedonly = FALSE.
-
?TADA_FlagSpeciation
-
When clean = “none”, this function adds the following column to your dataframe: TADA.MethodSpeciation.Flag. This column flags each TADA.CharacteristicName and MethodSpeciationName combination in your dataframe as either “NonStandardized”,
“Invalid”, or “Valid”.
When clean = “invalid_only”, only “Invalid” rows are removed from the dataframe. Default is clean = “invalid_only”.
When clean = “nonstandardized_only”, only “NonStandardized” rows are removed from the dataframe.
When clean = “both”, “Invalid” and “NonStandardized” rows are removed from the dataframe.
When clean = “none”, no rows are removed from the dataframe.
When flaggedonly = TRUE, the dataframe is filtered to only the rows flagged as “Invalid” or “NonStandardized”; default is flaggedonly = FALSE.
-
-
?TADA_FlagResultUnit
When clean = FALSE, the following column will be added to your dataframe: TADA.ResultUnit.Flag. This column flags each TADA.CharacteristicName, TADA.ActivityMediaName, and TADA.ResultMeasure.MeasureUnitCode combination in your dataframe as either “NonStandardized”, “Invalid”, or “Valid”.
When clean = TRUE, “Invalid” rows are removed from the dataframe and no column will be appended.
When flaggedonly = TRUE, the dataframe is filtered to only the rows flagged as “Invalid”; default is flaggedonly = FALSE.
-
?TADA_FlagFraction
- When clean = FALSE, this function adds the following column to your dataframe: TADA.SampleFraction.Flag. This column flags each combination in your dataframe as either “NonStandardized”, “Invalid”, or “Valid”.
- When clean = TRUE, “Invalid” rows are removed from the dataframe and no column will be appended.
- When flaggedonly = TRUE, the dataframe is filtered to only the rows flagged as “Invalid”; default is flaggedonly = FALSE.
R5ProfileClean2 <- TADA_FlagMethod(R5ProfileClean1, clean = TRUE)
R5ProfileClean2 <- TADA_FlagFraction(R5ProfileClean2, clean = TRUE)
R5ProfileClean2 <- TADA_FlagSpeciation(R5ProfileClean2, clean = "invalid_only")
R5ProfileClean2 <- TADA_FlagResultUnit(R5ProfileClean2, clean = "invalid_only")Question 6: How many rows were removed due to the WQX QAQC service functions? What might be a helpful first step in investigating why these removals occurred?
# Determine number of rows removed
R5ProfileClean1_n <- nrow(R5ProfileClean1)
R5ProfileClean2_n <- nrow(R5ProfileClean2)
rows_removed_n <- R5ProfileClean1_n - R5ProfileClean2_nAnswer: 37 rows were removed from the data set as a result of TADA_FlagMethod, TADA_FlagFraction, TADA_FlagSpeciation, and TADA_FlagResultUnit. To learn more about specific methods, fractions, speciation, or result units that were flagged for removal, you could re-run the flag functions with clean = FALSE and filter by flag types.
WQX national upper and lower thresholds
Run the following code to flag or remove results that are above or below the national upper and lower bound for each characteristic and unit combination. See documentation for more details:
-
?TADA_FlagAboveThreshold
When clean = FALSE, the following column is added to your dataframe: TADA.ResultValueAboveUpperThreshold.Flag. This column flags rows with data that are above the upper WQX threshold. The default is clean = FALSE.
When clean = TRUE, data that is above the upper WQX threshold is removed from the dataframe.
When flaggedonly = TRUE, the dataframe is filtered to only the rows flagged as above the upper WQX threshold; default is flaggedonly = FALSE.
-
?TADA_FlagBelowThreshold
When clean = FALSE, the following column is added to your dataframe: TADA.ResultValueBelowLowerThreshold.Flag. This column flags rows with data that are below the lower WQX threshold. The default is clean = FALSE.
When clean = TRUE, data that is below the lower WQX threshold is removed from the dataframe.
When flaggedonly = TRUE, the dataframe is filtered to only the rows flagged as below the lower WQX threshold; default is flaggedonly = FALSE.
R5ProfileClean3 <- TADA_FlagAboveThreshold(R5ProfileClean2, clean = TRUE)
R5ProfileClean3 <- TADA_FlagBelowThreshold(R5ProfileClean3, clean = TRUE)
#> [1] "No data below the WQX Lower Threshold were found in your dataframe. Returning the input dataframe with TADA.ResultValueBelowLowerThreshold.Flag column for tracking."Question 7: How many results were removed due to the WQX upper and lower thresholds?
# Determine number of rows removed
R5ProfileClean2_n <- nrow(R5ProfileClean2)
R5ProfileClean3_n <- nrow(R5ProfileClean3)
threshold_removed_n <- R5ProfileClean2_n - R5ProfileClean3_nAnswer: 255 results were removed due to falling outside the WQX thresholds.
Potential duplicates
Sometimes multiple organizations submit the exact same data to Water Quality Portal (WQP). Let’s start by looking at the submitting organizations for this data set and their relative contributions.
# Create data frame with result counts by organization
FieldValues_Orgs <- TADA_FieldValuesTable(R5ProfileClean3, field = "OrganizationFormalName")Question 8: How many organizations submitted data? Which organization submitted the most results? Which organization submitted the fewest results?
Answer: 15 unique organizations submitted data. Wisconsin Department of Natural Resources submitted the most results (n = 134983). Ohio River Valley Water Sanitation Commission (ORSANCO) submitted the fewest results (n = 2).
Organizations occasionally submit the same data multiple times to the Portal. The following functions check for and identify data that may be duplicates based on date, time, characteristic, result value, and a distance buffer. Each pair or group of potential duplicate rows is flagged with a unique ID. For more information, review the documentation by entering the following into the console:
- ?TADA_FindPotentialDuplicatesMultipleOrgs
- ?TADA_FindPotentialDuplicatesSingleOrg
# Flag potential duplicates from multiple organizations
R5ProfileClean3 <- TADA_FindPotentialDuplicatesMultipleOrgs(R5ProfileClean3,
dist_buffer = 100,
org_hierarchy = "none"
)
#> [1] "112 potentially duplicated results found in dataset. These have been placed into duplicate groups in the TADA.MultipleOrgDupGroupID column and the TADA.MultipleOrgDuplicate column is set to 'Y' (yes). If you provided an organization hierarchy, the result with the lowest ranked organization identifier was selected as the representative result in the TADA.ResultSelectedMultipleOrgs (this column is set to 'Y' for all results either selected or not considered duplicates)."
# Flag potential duplicates from a single organization
R5ProfileClean3 <- TADA_FindPotentialDuplicatesSingleOrg(R5ProfileClean3)
#> [1] "17797 groups of potentially duplicated results found in dataset. These have been placed into duplicate groups in the TADA.SingleOrgDupGroupID column and the function randomly selected one result from each group to represent a single, unduplicated value. Selected values are indicated in the TADA.SingleOrgDup.Flag as 'Unique', while duplicates are flagged as 'Duplicate' for easy filtering."
# Remove multiple and single org duplicates (random selection of single result)
R5ProfileClean4 <- R5ProfileClean3 %>%
dplyr::filter(
TADA.ResultSelectedMultipleOrgs == "Y",
TADA.SingleOrgDup.Flag == "Unique"
)Question 9: How many total duplicate results were removed? How could you prioritize retaining results from your organization when duplicate results are identified?
# Determine how many duplicate results were removed
R5ProfileClean3_n <- nrow(R5ProfileClean3)
R5ProfileClean4_n <- nrow(R5ProfileClean4)
duplicate_removed_n <- R5ProfileClean3_n - R5ProfileClean4_nAnswer: 112654 duplicates were removed by TADA_FindPotentialDuplicatesMultipleOrgs and TADA_FindPotentialDuplicatesSingleOrg. To prioritize results from a particular agency or agencies see documentation for ?TADA_FindDuplicatesMultipleOrgs for information on using the org_hierarchy argument.
Full Dataframe Filtering
In this section a TADA user can review the unique values in specific fields and may choose to remove data with particular values.
To start, review the list of common fields used for filtering, and the number of unique values in each field using the TADA_FieldCounts function.
This function returns counts for you the entire data frame for each of the following fields (if populated, columns that are populated only with NA’s are not included in the output):
ActivityTypeCode
TADA.ActivityMediaName
ActivityMediaSubdivisionName
ActivityCommentText
MonitoringLocationTypeName
StateCode
OrganizationFormalName
TADA.CharacteristicName
HydrologicCondition
HydrologicEvent
BiologicalIntentName
MeasureQualifierCode
ActivityGroup
AssemblageSampledName
ProjectName
CharacteristicNameUserSupplied
DetectionQuantitationLimitTypeName
SampleTissueAnatomyName
LaboratoryName
# Create table of field counts
FieldCounts_Table_R5 <- TADA_FieldCounts(R5ProfileClean4)The ActivityTypeCode field has multiple unique values. Before we remove the QC samples/measurements from this dataset to prepare for analyses, lets review flagged Quality Control (QC) values using the TADA_FindQCActivities function, which a TADA.ActivityType.Flag column.
For example, the new QC_replicate flag in TADA.ActivityType.Flag column indicates that the flagged rows include any of the following replicate values: Quality Control Field Replicate Habitat Assessment, Quality Control Field Replicate Msr/Obs, Quality Control Field Replicate Portable Data Logger, Quality Control Field Replicate Sample-Composite and Quality Control Sample-Field Replicate.
See WQX domain file to review all the ActivityTypeCode allowable values: https://cdx.epa.gov/wqx/download/DomainValues/ActivityType.CSV
Other functions, such as TADA_PairReplicates which identifies replicates and their original (parent) samples, may also be useful when reviewing QC results. For now, let’s simply remove QC samples/measurements from the dataframe.
For more information on TADA_PairReplicates and suggestions for handling paired replicates (outside the TADA package) see TADA Module 1 or enter “?TADA_PairReplicates” in the console.
# Remove flagged QC samples using the TADA_FindQCActivities function:
R5ProfileClean5 <- TADA_FindQCActivities(R5ProfileClean4,
clean = TRUE
)
#> [1] "Quality control samples have been removed or were not present in the input dataframe. Returning dataframe with TADA.ActivityType.Flag column for tracking."
# Generate pie chart for ActivityTypeCode
ActTypCod_Pie <- TADA_FieldValuesPie(R5ProfileClean5, "ActivityTypeCode")
ActTypCod_Pie
Question 10: How many ActivityTypeCodes are present after removing QC samples? Which ActivityTypeCode is the most common?
# Number of ActivityTypeCodes
activity_type_code_n <- length(unique(R5ProfileClean5$ActivityTypeCode))
# Most common Activity Type Code
most_common_activity <- TADA_FieldValuesTable(R5ProfileClean5, "ActivityTypeCode") %>%
dplyr::slice_max(Count)Answer: There are 3 unique ActivityTypeCodes present after removing QC samples. Field Msr/Obs is the most common ActivityTypeCode (n = 25215).
We’ve completed our review of the ActivityTypeCode. Now, let’s move on to a different field and see if there are any values that we want to remove. In this next example, there are multiple MeasureQualifierCode values to review.
MQC_Pie <- TADA_FieldValuesPie(R5ProfileClean5, "MeasureQualifierCode")
MQC_Pie
MeasureQualifierCode definitions are available here.
In this example, we show how to use the function TADA_FlagMeasureQualifierCode to add MeasureQualifierCode definitions and flag and/or remove rows with specific codes under MeasureQualifierCode that are categorized as “SUSPECT”.
See ?TADA_FlagMeasureQualifierCode for more information.
# Flag only
Review_R5ProfileClean5 <- TADA_FlagMeasureQualifierCode(R5ProfileClean5,
clean = FALSE,
flaggedonly = TRUE,
define = TRUE
)
# Review number of suspect results
suspect_n <- Review_R5ProfileClean5 %>%
dplyr::filter(TADA.MeasureQualifierCode.Flag == "SUSPECT") %>%
nrow()
# Run function with clean = TRUE
R5ProfileClean5 <- TADA_FlagMeasureQualifierCode(R5ProfileClean5,
clean = TRUE
)Question 11: How many results were identified as “SUSPECT” based on the MeasureQualifierCode?
Answer: 0 results were identified as suspect based on the MeasureQualifierCode.
Censored data
Censored data are measurements for which the true value is not known, but we can estimate the value based on lower or upper detection conditions and limit types. TADA fills missing TADA.ResultMeasureValue and TADA.ResultMeasure.MeasureUnitCode values with values and units from TADA.DetectionQuantitationLimitMeasure.MeasureValue and TADA.DetectionQuantitationLimitMeasure.MeasureUnitCode, respectively, using the TADA_AutoClean function. In other words, detection limit information is copied and pasted into the result value column when the original value is NA and detection limit information is available. The two columns TADA focuses on to define and flag censored data are ResultDetectionConditionText and DetectionQuantitationLimitTypeName.
The TADA package currently has functions that summarize censored data incidence in the dataset and perform simple substitutions of censored data values, including x times the detection limit and random selection of a value between 0 and the detection limit. The user may specify the methods used for non-detects and over-detects separately in the input to the TADA_SimpleCensoredMethods function.
All censored data functions depend first on the TADA_IDCensoredData utility function, which assigns a TADA.CensoredData.Flag to all data records and identifies over-detects from non-detects using the ResultDetectionConditionText and DetectionQuantitationLimitTypeName. This utility function is automatically run within the TADA_AutoClean function and produces the TADA.CensoredData.Flag column. All records receive one of the following classifications: Uncensored; Non-Detect; Over-Detect; Other Condition/Limit Populated; Conflict between Condition and Limit; Detection condition or detection limit is not documented in TADA; or Detection condition is missing and required for censored data ID.
The TADA_SimpleCensoredMethods function also adds a TADA.MeasureQualifierCode.Def column which contains the MeasureQualiferCode concatenated with the WQX definition for each qualifier code. This provides additional information to the user which may assist in deciding which records to retain for analysis.
The next step we take in this example is to perform simple conversions to the censored data in the dataset: we keep over-detects as is (no conversion made) and convert non-detect values to 0.5 times the detection limit (half the detection limit). Please review ?TADA_Stats and ?TADA_SimpleCensoredMethods for more information.
R5ProfileClean5 <- TADA_SimpleCensoredMethods(R5ProfileClean5,
nd_method = "multiplier",
nd_multiplier = 0.5,
od_method = "as-is",
od_multiplier = "null"
)
#> [1] "TADA_IDCensoredData: 18 records in supplied dataset have conflicting detection condition and detection limit type information. These records will not be included in detection limit handling calculations."Question 12: How many NAs in the TADA.ResultMeasureValue column remain after running TADA_SimpleCensoredMethods? What is the difference between the values in the MeasureQualifierCode and TADA.MeasureQualifierCode.Def columns?
# Determine number of NA results remaining
result_na_n <- sum(is.na(R5ProfileClean5$TADA.ResultMeasureValue))
# Show unique values of MeasureQualifierCode and TADA.MeasureQualifierCode.Def
mqc <- R5ProfileClean5 %>%
dplyr::select(MeasureQualifierCode, TADA.MeasureQualifierCode.Def) %>%
unique()Answer: There are 1374 TADA.ResultMeasureValue NAs remaining in the data set. The MeasureQualifierCode column contains qualifier code abbreviations. TADA.MeasureQualifierCode.Def contains qualifier code abbreviations concatenated with their definition.
Next, we need to filter down to only numeric data. Remove data where the TADA.ResultMeasureValueDataTypes.Flag = “Text” or “NA - Not Available”. You can also remove any columns not required for the TADA workflow that contain only NAs. The TADA_AutoFilter function removes non-numeric data and optional columns containing only NAs.
# Removes rows where the result value is not numeric. Specifically, removes rows with "Text" or "NA - Not Available" in the TADA.ResultMeasureValueDataTypes.Flag column, or NA in the TADA.ResultMeasureValue column. Removes optional columns containing only NAs.
R5ProfileClean6 <- TADA_AutoFilter(R5ProfileClean5)
#> [1] "TADA_Autofilter: removing columns not required for TADA workflow if they contain only NAs."
#> [1] "The following column(s) were removed as they contained only NAs: ActivityDepthAltitudeReferencePointText, SampleAquifer, ResultParticleSizeBasisText, USGSPCode, BinaryObjectFileName, BinaryObjectFileTypeCode, LabSamplePreparationUrl, HorizontalAccuracyMeasure.MeasureValue, HorizontalAccuracyMeasure.MeasureUnitCode, VerticalAccuracyMeasure.MeasureValue, VerticalAccuracyMeasure.MeasureUnitCode, FormationTypeText, ProjectMonitoringLocationWeightingUrl, DrainageAreaMeasure.MeasureValue, DrainageAreaMeasure.MeasureUnitCode, ContributingDrainageAreaMeasure.MeasureValue and ContributingDrainageAreaMeasure.MeasureUnitCode."
#> [1] "TADA_Autofilter: checking required columns for non-NA values."
#> [1] "TADA Required column(s) SubjectTaxonomicName, SampleTissueAnatomyName, ResultDepthHeightMeasure.MeasureValue, TADA.ResultDepthHeightMeasure.MeasureValue, ResultDepthHeightMeasure.MeasureUnitCode, TADA.ResultDepthHeightMeasure.MeasureUnitCode, ResultDepthAltitudeReferencePointText, ActivityTopDepthHeightMeasure.MeasureValue, TADA.ActivityTopDepthHeightMeasure.MeasureValue, ActivityTopDepthHeightMeasure.MeasureUnitCode, TADA.ActivityTopDepthHeightMeasure.MeasureUnitCode, ActivityBottomDepthHeightMeasure.MeasureValue, TADA.ActivityBottomDepthHeightMeasure.MeasureValue, ActivityBottomDepthHeightMeasure.MeasureUnitCode, TADA.ActivityBottomDepthHeightMeasure.MeasureUnitCode, ResultFileUrl, ResultAnalyticalMethod.MethodUrl, HydrologicCondition, HydrologicEvent, DataQuality.PrecisionValue, DataQuality.BiasValue, DataQuality.ConfidenceIntervalValue, DataQuality.UpperConfidenceLimitValue, DataQuality.LowerConfidenceLimitValue, ResultLaboratoryCommentText, AquiferName, AquiferTypeName, LocalAqfrName, ConstructionDateText, WellDepthMeasure.MeasureValue, WellDepthMeasure.MeasureUnitCode, WellHoleDepthMeasure.MeasureValue and WellHoleDepthMeasure.MeasureUnitCode contain only NA values. This may impact other TADA functions."
#> [1] "Function removed 1374 results. These results are either text or NA and cannot be plotted or represent quality control activities (not routine samples or measurements)."Question 13: How many NAs remain in the data set for TADA.ResultMeasureValue after running TADA_Autofilter? How many non-numeric values in TADA.ResultMeasureValueDataTypes.Flag remain in the data set? How many rows were removed from the data set?
result_nas_n <- sum(is.na(R5ProfileClean6$TADA.ResultMeasureValue))
non_numeric_n <- sum(!R5ProfileClean6$TADA.ResultMeasureValueDataTypes.Flag %in% c("Numeric", "Result Value/Unit Estimated from Detection Limit", "Result Value/Unit Copied from Detection Limit"))
R5ProfileClean5_n <- nrow(R5ProfileClean5)
R5ProfileClean6_n <- nrow(R5ProfileClean6)
autofilter_removed_n <- R5ProfileClean5_n - R5ProfileClean6_nAnswer: There are 0 results remaining where TADA.ResultMeasureValue is equal to NA after running TADA_AutoFilter. There are 0 results remaining where TADA.ResultMeasureValueDataTypes is non-numeric. 1374 results where removed from the data set to ensure that only numeric results are retained.
Convert synonymous characteristic, fraction, speciation, and unit values to a consistent convention based on user-defined/TADA standards
The TADA_GetSynonymRef function generates a synonym reference table that is specific to the input dataframe. Users can review how their input data relates to standard TADA values for the following elements:
TADA.CharacteristicName
TADA.ResultSampleFractionText
TADA.MethodSpeciationName
TADA.ResultMeasure.MeasureUnitCode
Users can also edit the reference file to meet their needs if desired. The download argument can be used to save the harmonization file to your current working directory when download = TRUE, the default is download = FALSE.
The TADA_HarmonizeSynonyms function then compares the input dataframe to the TADA Synonym Reference Table and makes conversions where target characteristics/fractions/speciations/units are provided. This function also appends a column called TADA.Harmonized.Flag, indicating which results had metadata changed/converted in this function. The purpose of this function is to make similar data consistent and therefore easier to compare and analyze.
Here are some examples of how the TADA_HarmonizeSynonyms can be useful:
TADA.ResultSampleFractionText specifies forms of constituents. In some cases, a single TADA.CharacteristicName will have both “Total” and “Dissolved” forms specified, which should not be combined. In these cases, each TADA.CharacteristicName and TADA.ResultSampleFractionText combination is given a different identifier. This identifier can be used later on to identify comparable data groups (TADA.ComparableDataIdentifier) for calculating statistics and creating figures for each combination.
Some variables have different names but represent the same constituent (e.g., “Total Kjeldahl nitrogen (Organic N & NH3)” and “Kjeldahl nitrogen”). The TADA_HarmonizeSynonyms function gives a consistent name (and identifier) to synonyms.
# Get harmonization reference table
UniqueHarmonizationRef <- TADA_GetSynonymRef(R5ProfileClean6)
# Filter out records without target to display in table
UniqueHarmonizationRef_no_na <- UniqueHarmonizationRef %>%
dplyr::filter(is.na(Target.TADA.CharacteristicName))
# Harominze data set
R5ProfileClean7 <- TADA_HarmonizeSynonyms(R5ProfileClean6,
ref = UniqueHarmonizationRef
)The table above shows the first 100 rows of the UniqueHarmonizationRef table.
Total Nitrogen and Total Phosphorus Calculations
This section covers summing nutrient subspecies to estimate total nitrogen and phosphorus. This can be challenging because some subspecies/compounds overlap in total nutrient calculations. Thus, TADA_CalculateTotalNP uses the Nutrient Aggregation logic to add subspecies to obtain a total. TADA adds one more equation to the mix: total particulate nitrogen + total dissolved nitrogen. The function uses as many subspecies as possible to calculate a total for each given site, date, and depth group, but it will estimate total nitrogen with whatever subspecies are present. This function creates NEW total nutrient measurements (total nitrogen unfiltered as N and total phosphorus unfiltered as P) and adds them to the dataframe.
Users can use the default summation worksheet (see TADA_GetNutrientSummationRef) or customize it. The function also requires a daily aggregation value, either minimum, maximum, or mean. The default is ‘max’, which means that if multiple measurements of the same subspecies-fraction-speciation-unit occur on the same day at the same site and depth, the function will pick the maximum value to use for summation.
R5ProfileClean8 <- TADA_CalculateTotalNP(R5ProfileClean7, daily_agg = "max")
#> [1] "Aggregation results:"
#>
#> No aggregation needed Selected as max aggregate value
#> 7242 84Question 14: How many new Total Nitrogen or Total Phosphorus rows were added? Can users customize nutrient summation?
R5ProfileClean7_n <- nrow(R5ProfileClean7)
R5ProfileClean8_n <- nrow(R5ProfileClean8)
nutrient_n <- R5ProfileClean8_n - R5ProfileClean7_n
total_N_P_subset <- R5ProfileClean8 %>%
dplyr::filter(TADA.NutrientSummation.Flag %in% c("Nutrient summation from one or more subspecies.", "Nutrient summation from one subspecies.")) %>%
dplyr::select(TADA.CharacteristicName, TADA.ResultMeasureValue, TADA.ResultMeasure.MeasureUnitCode, TADA.NutrientSummation.Flag, TADA.NutrientSummationEquation, TADA.NutrientSummationGroup)The table above shows the first 100 rows added by calculating Total N and P. Columns included in the table are TADA.CharacteristicName, TADA.ResultMeasureValue, TADA.ResultMeasure.MeasureUnitCode, TADA.NutrientSummation.Flag, TADA.NutrientSummationEquation, and TADA.NutrientSummationGroup.
Answer: 597 Total Nitrogen or Total Phosphorus rows were added. Users can review the summation worksheet (TADA_GetNutrientSummationRef) and update it to meet their requirements.
Parameter Level Filtering
In this section, you can select a single parameter, and review the unique values in specified fields. You may then choose to remove particular values by filtering.
To start, review the list of parameters in the dataframe using the TADA_FieldValuesTable function.
Char_Pie <- FieldValuesTable_Chars <- TADA_FieldValuesTable(R5ProfileClean8, field = "TADA.CharacteristicName")
Char_Pie
#> Value
#> 1 TEMPERATURE
#> 2 TOTAL PHOSPHORUS, MIXED FORMS
#> 3 TRANSPARENCY, TUBE WITH DISK
#> 4 DISSOLVED OXYGEN (DO)
#> 5 SPECIFIC CONDUCTANCE
#> 6 PH
#> 7 TOTAL SUSPENDED SOLIDS
#> 8 TOTAL NITROGEN, MIXED FORMS
#> 9 TOTAL KJELDAHL NITROGEN (ORGANIC N & NH3)
#> 10 DISSOLVED OXYGEN SATURATION
#> 11 NITRATE + NITRITE
#> 12 CHLORIDE
#> 13 ORTHOPHOSPHATE
#> 14 AMMONIA
#> 15 TURBIDITY
#> 16 CADMIUM
#> 17 CHROMIUM
#> 18 COPPER
#> 19 LEAD
#> 20 NICKEL
#> 21 ZINC
#> 22 ORGANIC CARBON
#> 23 MAGNESIUM
#> 24 CALCIUM
#> 25 ARSENIC
#> 26 SELENIUM
#> 27 SULFATE
#> 28 POTASSIUM
#> 29 SODIUM
#> 30 IRON
#> 31 STRONTIUM
#> 32 ALUMINUM
#> 33 BORON
#> 34 ESCHERICHIA COLI
#> 35 MANGANESE
#> 36 BARIUM
#> 37 SILVER
#> 38 COBALT
#> 39 VANADIUM
#> 40 BERYLLIUM
#> 41 HEIGHT, GAGE
#> 42 DISTANCE FROM/TO
#> 43 NITRATE
#> 44 TOTAL DISSOLVED SOLIDS
#> 45 HARDNESS, CARBONATE
#> 46 STREAM STAGE
#> 47 PHOSPHATE
#> 48 VOLATILE SUSPENDED SOLIDS
#> 49 FECAL COLIFORM
#> 50 HARDNESS, CA, MG
#> 51 CHEMICAL OXYGEN DEMAND
#> 52 TEMPERATURE, AIR
#> 53 FLOW
#> 54 TEMPERATURE, SAMPLE
#> 55 CYANIDE, AVAILABLE
#> 56 PHENOLS
#> 57 PHOSPHORUS
#> 58 TOTAL SOLIDS
#> 59 CLOUD COVER
#> 60 NITRITE
#> 61 BIOCHEMICAL OXYGEN DEMAND, STANDARD CONDITIONS
#> 62 CHLOROPHYLL A
#> 63 TOTAL COLIFORM
#> 64 INORGANIC CARBON
#> 65 OXIDATION REDUCTION POTENTIAL (ORP)
#> 66 CHLOROPHYLL A, CORRECTED FOR PHEOPHYTIN
#> 67 FLUORIDE
#> 68 SALINITY
#> 69 CHLOROPHYLL C
#> 70 PHEOPHYTIN A
#> 71 CHLOROPHYLL A, UNCORRECTED FOR PHEOPHYTIN
#> 72 CHLOROPHYLL B
#> 73 ACETOCHLOR
#> 74 ALACHLOR
#> 75 AMMONIUM
#> 76 ATRAZINE
#> 77 CARBONACEOUS BIOCHEMICAL OXYGEN DEMAND, STANDARD CONDITIONS
#> 78 CHLORPYRIFOS
#> 79 CONDUCTIVITY
#> 80 CYANAZINE
#> 81 DEPTH, BOTTOM
#> 82 MERCURY
#> 83 METHOXYCHLOR
#> 84 METOLACHLOR
#> 85 MOLYBDENUM
#> 86 PENDIMETHALIN
#> 87 SIMAZINE
#> 88 THALLIUM
#> 89 TITANIUM
#> 90 TRANSPARENCY, SECCHI TUBE WITH DISK
#> 91 TRIFLURALIN
#> 92 ALDRIN
#> 93 CHLOROPHYLL A, FREE OF PHEOPHYTIN
#> 94 CIS-CHLORDANE
#> 95 DIELDRIN
#> 96 ENDRIN
#> 97 HEPTACHLOR
#> 98 HEPTACHLOR EPOXIDE
#> 99 HEXACHLOROBENZENE
#> 100 LINDANE
#> 101 P,P'-DDT
#> 102 TRANS-CHLORDANE
#> 103 ACIFLUORFEN
#> 104 CYANIDE
#> 105 DIAZINON
#> 106 DICAMBA
#> 107 FONOFOS
#> 108 MALATHION
#> 109 METHYL PARATHION
#> 110 NITROGEN
#> 111 PHORATE
#> 112 PICLORAM
#> 113 S-ETHYL DIPROPYLTHIOCARBAMATE
#> 114 SILVEX
#> 115 TERBUFOS
#> 116 TOTAL HARDNESS
#> 117 TOTAL VOLATILE SOLIDS
#> 118 VELOCITY - STREAM
#> 119 .ALPHA.-HEXACHLOROCYCLOHEXANE
#> 120 2-CHLORO-4-ISOPROPYLAMINO-6-AMINO-S-TRIAZINE
#> 121 2-CHORO-6-ETHYLAMINO-4-AMINO-S-TRIAZINE
#> 122 BUTYLATE
#> 123 CAPTAN
#> 124 CLOMAZONE
#> 125 DALAPON
#> 126 DINOSEB
#> 127 GLYPHOSATE
#> 128 METRIBUZIN
#> 129 P,P'-DDD
#> 130 P,P'-DDE
#> 131 PARATHION
#> 132 PENTACHLOROPHENOL
#> 133 POLYCHLORINATED BIPHENYLS
#> 134 PROPACHLOR
#> 135 SUSPENDED SEDIMENT CONCENTRATION (SSC)
#> 136 TOXAPHENE
#> 137 2,4-D
#> 138 APPARENT COLOR
#> 139 BENZO[A]PYRENE
#> 140 CIS-NONACHLOR
#> 141 DI(2-ETHYLHEXYL) ADIPATE
#> 142 DI(2-ETHYLHEXYL) PHTHALATE
#> 143 HEXACHLOROCYCLOPENTADIENE
#> 144 OXYCHLORDANE
#> 145 TRANS-NONACHLOR
#> 146 TRUE COLOR
#> 147 VELOCITY-DISCHARGE
#> 148 .LAMBDA.-CYHALOTHRIN
#> 149 2-CHLORO-4,6-DIAMINO-S-TRIAZINE
#> 150 2-HYDROXYATRAZINE
#> 151 2,4-DB
#> 152 2,4,5-T
#> 153 ACETAMIPRID
#> 154 ACETOCHLOR ESA
#> 155 ACETOCHLOR OA
#> 156 AFIDOPYROPEN
#> 157 ALACHLOR ESA
#> 158 ALACHLOR OA
#> 159 ALDICARB SULFONE
#> 160 ALDICARB SULFOXIDE
#> 161 AMINOPYRALID
#> 162 AZOXYSTROBIN
#> 163 BENFLURALIN
#> 164 BENSULFURON-METHYL
#> 165 BENSULIDE
#> 166 BENTAZON
#> 167 BENZENEPROPANENITRILE, .ALPHA.-(CYCLOPROPYLCARBONYL)- 2-(METHYLSULFONYL)-.BETA.- OXO-4-(TRIFLUOROMETHYL)-
#> 168 BENZOVINDIFLUPYR
#> 169 BETA CYPERMETHRIN
#> 170 BICYCLOPYRONE
#> 171 BICYCLOPYRONE SYN503870
#> 172 BIFENTHRIN
#> 173 BOSCALID
#> 174 BROMACIL
#> 175 BROMOXYNIL
#> 176 CALCIUM CARBONATE
#> 177 CARBARYL
#> 178 CARBENDAZIM
#> 179 CARBOFURAN
#> 180 CHLORANTRANILIPROLE
#> 181 CHLORIMURON-ETHYL
#> 182 CHLOROTHALONIL
#> 183 CHLORPYRIFOS O.A.
#> 184 CLETHODIM SULFONE
#> 185 CLETHODIM SULFOXIDE
#> 186 CLOPYRALID
#> 187 CLOTHIANIDIN
#> 188 CYANAZINE ACID
#> 189 CYANAZINE AMIDE
#> 190 CYANTRANILIPROLE
#> 191 CYFLUTHRIN
#> 192 DEETHYLCYANAZINE
#> 193 DEETHYLCYANAZINE ACID
#> 194 DEETHYLCYANAZINE AMIDE
#> 195 DESMETHYLNORFLURAZON
#> 196 DIAZOXON
#> 197 DICHLOBENIL
#> 198 DICHLORPROP
#> 199 DICHLORVOS
#> 200 DICROTOPHOS
#> 201 DIFENOCONAZOLE
#> 202 DIMETHENAMID
#> 203 DIMETHENAMID ESA
#> 204 DIMETHENAMID OA
#> 205 DIMETHOATE
#> 206 DINOTEFURAN
#> 207 DISULFOTON
#> 208 DISULFOTON SULFONE
#> 209 DIURON
#> 210 ESFENVALERATE
#> 211 ETHALFLURALIN
#> 212 ETHOFUMESATE
#> 213 FLUFENACET OA
#> 214 FLUMETSULAM
#> 215 FLUPYRADIFURONE
#> 216 FLUTIANIL
#> 217 FLUTIANIL OC 56574
#> 218 FLUTIANIL OC 56635
#> 219 FLUTRIAFOL
#> 220 FLUXAPYROXAD
#> 221 FOMESAFEN
#> 222 HALAUXIFEN ACID
#> 223 HALAUXIFEN-METHYL
#> 224 HALOSULFURON-METHYL
#> 225 HEXAZINONE
#> 226 IMAZAMETHABENZ ACID
#> 227 IMAZAMETHABENZ-METHYL
#> 228 IMAZAMOX
#> 229 IMAZAPIC
#> 230 IMAZAPYR
#> 231 IMAZAQUIN
#> 232 IMAZETHAPYR
#> 233 IMIDACLOPRID
#> 234 IMIDACLOPRID-OLEFIN
#> 235 IMIDACLOPRID-UREA
#> 236 ISOXAFLUTOLE
#> 237 LINURON
#> 238 MANDESTROBIN
#> 239 MCPA
#> 240 MCPB
#> 241 MECOPROP
#> 242 MESOTRIONE
#> 243 METALAXYL
#> 244 METHYL PARAOXON
#> 245 METOLACHLOR ESA
#> 246 METOLACHLOR OA
#> 247 METRIBUZIN DA
#> 248 METRIBUZIN DADK
#> 249 METRIBUZIN DK
#> 250 METSULFURON-METHYL
#> 251 MOMFLUOROTHRIN
#> 252 MYCLOBUTANIL
#> 253 NICOSULFURON
#> 254 NORFLURAZON
#> 255 OXADIAZON
#> 256 OXATHIAPIPROLIN
#> 257 OXYDEMETON-METHYL
#> 258 PHOSTEBUPIRIM
#> 259 PICOXYSTROBIN
#> 260 PROMETON
#> 261 PROMETRYN
#> 262 PROPACHLOR ESA
#> 263 PROPACHLOR OA
#> 264 PROPAZINE
#> 265 PROPICONAZOLE
#> 266 PYDIFLUMETOFEN
#> 267 PYRACLOSTROBIN
#> 268 PYROXASULFONE
#> 269 SAFLUFENACIL
#> 270 SEDAXANE
#> 271 SIDURON
#> 272 SULFENTRAZONE
#> 273 SULFOMETURON METHYL
#> 274 TEBUCONAZOLE
#> 275 TEMBOTRIONE
#> 276 TETRACONAZOLE
#> 277 THIACLOPRID
#> 278 THIAMETHOXAM
#> 279 THIFENSULFURON-METHYL
#> 280 THIOBENCARB
#> 281 TOLFENPYRAD
#> 282 TOLPYRALATE
#> 283 TRIALLATE
#> 284 TRIASULFURON
#> 285 TRICLOPYR
#> Count
#> 1 22200
#> 2 630
#> 3 418
#> 4 414
#> 5 389
#> 6 382
#> 7 304
#> 8 279
#> 9 234
#> 10 199
#> 11 166
#> 12 138
#> 13 123
#> 14 122
#> 15 104
#> 16 99
#> 17 99
#> 18 99
#> 19 99
#> 20 99
#> 21 99
#> 22 98
#> 23 96
#> 24 95
#> 25 93
#> 26 93
#> 27 93
#> 28 89
#> 29 89
#> 30 87
#> 31 87
#> 32 86
#> 33 81
#> 34 76
#> 35 71
#> 36 66
#> 37 66
#> 38 62
#> 39 62
#> 40 60
#> 41 56
#> 42 50
#> 43 49
#> 44 49
#> 45 47
#> 46 45
#> 47 44
#> 48 43
#> 49 40
#> 50 36
#> 51 34
#> 52 32
#> 53 31
#> 54 29
#> 55 28
#> 56 28
#> 57 28
#> 58 28
#> 59 27
#> 60 26
#> 61 22
#> 62 16
#> 63 16
#> 64 15
#> 65 13
#> 66 12
#> 67 12
#> 68 10
#> 69 9
#> 70 9
#> 71 8
#> 72 8
#> 73 6
#> 74 6
#> 75 6
#> 76 6
#> 77 6
#> 78 6
#> 79 6
#> 80 6
#> 81 6
#> 82 6
#> 83 6
#> 84 6
#> 85 6
#> 86 6
#> 87 6
#> 88 6
#> 89 6
#> 90 6
#> 91 6
#> 92 5
#> 93 5
#> 94 5
#> 95 5
#> 96 5
#> 97 5
#> 98 5
#> 99 5
#> 100 5
#> 101 5
#> 102 5
#> 103 4
#> 104 4
#> 105 4
#> 106 4
#> 107 4
#> 108 4
#> 109 4
#> 110 4
#> 111 4
#> 112 4
#> 113 4
#> 114 4
#> 115 4
#> 116 4
#> 117 4
#> 118 4
#> 119 3
#> 120 3
#> 121 3
#> 122 3
#> 123 3
#> 124 3
#> 125 3
#> 126 3
#> 127 3
#> 128 3
#> 129 3
#> 130 3
#> 131 3
#> 132 3
#> 133 3
#> 134 3
#> 135 3
#> 136 3
#> 137 2
#> 138 2
#> 139 2
#> 140 2
#> 141 2
#> 142 2
#> 143 2
#> 144 2
#> 145 2
#> 146 2
#> 147 2
#> 148 1
#> 149 1
#> 150 1
#> 151 1
#> 152 1
#> 153 1
#> 154 1
#> 155 1
#> 156 1
#> 157 1
#> 158 1
#> 159 1
#> 160 1
#> 161 1
#> 162 1
#> 163 1
#> 164 1
#> 165 1
#> 166 1
#> 167 1
#> 168 1
#> 169 1
#> 170 1
#> 171 1
#> 172 1
#> 173 1
#> 174 1
#> 175 1
#> 176 1
#> 177 1
#> 178 1
#> 179 1
#> 180 1
#> 181 1
#> 182 1
#> 183 1
#> 184 1
#> 185 1
#> 186 1
#> 187 1
#> 188 1
#> 189 1
#> 190 1
#> 191 1
#> 192 1
#> 193 1
#> 194 1
#> 195 1
#> 196 1
#> 197 1
#> 198 1
#> 199 1
#> 200 1
#> 201 1
#> 202 1
#> 203 1
#> 204 1
#> 205 1
#> 206 1
#> 207 1
#> 208 1
#> 209 1
#> 210 1
#> 211 1
#> 212 1
#> 213 1
#> 214 1
#> 215 1
#> 216 1
#> 217 1
#> 218 1
#> 219 1
#> 220 1
#> 221 1
#> 222 1
#> 223 1
#> 224 1
#> 225 1
#> 226 1
#> 227 1
#> 228 1
#> 229 1
#> 230 1
#> 231 1
#> 232 1
#> 233 1
#> 234 1
#> 235 1
#> 236 1
#> 237 1
#> 238 1
#> 239 1
#> 240 1
#> 241 1
#> 242 1
#> 243 1
#> 244 1
#> 245 1
#> 246 1
#> 247 1
#> 248 1
#> 249 1
#> 250 1
#> 251 1
#> 252 1
#> 253 1
#> 254 1
#> 255 1
#> 256 1
#> 257 1
#> 258 1
#> 259 1
#> 260 1
#> 261 1
#> 262 1
#> 263 1
#> 264 1
#> 265 1
#> 266 1
#> 267 1
#> 268 1
#> 269 1
#> 270 1
#> 271 1
#> 272 1
#> 273 1
#> 274 1
#> 275 1
#> 276 1
#> 277 1
#> 278 1
#> 279 1
#> 280 1
#> 281 1
#> 282 1
#> 283 1
#> 284 1
#> 285 1Next, we can revisit the TADA_FieldCounts function at the characteristic level to review how many unique allowable values are included within each of the following fields:
ActivityCommentText
ActivityTypeCode
TADA.ActivityMediaName
ActivityMediaSubdivisionName
MeasureQualifierCode
MonitoringLocationTypeName
HydrologicCondition
HydrologicEvent
ResultStatusIdentifier
MethodQualifierTypeName
ResultCommentText
ResultLaboratoryCommentText
TADA.ResultMeasure.MeasureUnitCode
TADA.ResultSampleFractionText
ResultTemperatureBasisText
ResultValueTypeName
ResultWeightBasisText
SampleCollectionEquipmentName
LaboratoryName
MethodDescriptionText
ResultParticleSizeBasisText
SampleCollectionMethod.MethodIdentifier
SampleCollectionMethod.MethodIdentifierContext
SampleCollectionMethod.MethodName
DataQuality.BiasValue
MethodSpeciationName
ResultAnalyticalMethod.MethodName
ResultAnalyticalMethod.MethodIdentifier
ResultAnalyticalMethod.MethodIdentifierContext
AssemblageSampledName
DetectionQuantitationLimitTypeName
R5_FieldCounts_Chars <- TADA_FieldCounts(R5ProfileClean8, display = "most", characteristicName = "DISSOLVED OXYGEN (DO)")Selecting a parameter generates the list above, which is subset by the selected parameter. The list includes fields you may want to review, and the number of unique values in each field.
Next, choose a field from the list.
Review the WQX domain files for definitions: https://www.epa.gov/waterdata/storage-and-retrieval-and-water-quality-exchange-domain-services-and-downloads
Now, we’ll use TADA_FieldValuesTable and TADA_FieldValuesPie at the characteristic-level to review a column of interest. In this example we review values from the SampleCollectionMethod.MethodName field for Dissolved Oxygen (DO) results.
# C Create pie chart for SampleCollectionMethod.MethodName for Dissolved Oxygen (DO results)
DO_SCM_Pie <- TADA_FieldValuesPie(R5ProfileClean8, field = "SampleCollectionMethod.MethodName", characteristicName = "DISSOLVED OXYGEN (DO)")
DO_SCM_Pie
# Create table for SampleCollectionMethod.MethodName for Dissolved Oxygen (DO results)
FieldValuesTable_DO_scm <- TADA_FieldValuesTable(R5ProfileClean8, field = "SampleCollectionMethod.MethodName", characteristicName = "DISSOLVED OXYGEN (DO)")Generate a scatterplot with two characteristics.
# review unique identifiers
unique(R5ProfileClean8$TADA.ComparableDataIdentifier)
#> [1] "TOTAL PHOSPHORUS, MIXED FORMS_UNFILTERED_AS P_UG/L"
#> [2] "NITROGEN_TOTAL_NA_UG/L"
#> [3] "TOTAL SUSPENDED SOLIDS_NON-FILTERABLE (PARTICLE)_NA_MG/L"
#> [4] "DISSOLVED OXYGEN (DO)_NA_NA_MG/L"
#> [5] "DISSOLVED OXYGEN SATURATION_NA_NA_%"
#> [6] "TRANSPARENCY, TUBE WITH DISK_TOTAL_NA_CM"
#> [7] "PH_TOTAL_NA_NA"
#> [8] "SPECIFIC CONDUCTANCE_NA_NA_US/CM"
#> [9] "FLOW_NA_NA_CFS"
#> [10] "ESCHERICHIA COLI_NA_NA_#/100ML"
#> [11] "PHOSPHATE_UNFILTERED_AS P_UG/L"
#> [12] "TEMPERATURE_NA_NA_DEG C"
#> [13] "CLOUD COVER_NA_NA_%"
#> [14] "TURBIDITY_NA_NA_NTU"
#> [15] "ORTHOPHOSPHATE_FILTERED_AS P_UG/L"
#> [16] "TOTAL KJELDAHL NITROGEN (ORGANIC N & NH3)_UNFILTERED_AS N_MG/L"
#> [17] "SUSPENDED SEDIMENT CONCENTRATION (SSC)_TOTAL_NA_UG/L"
#> [18] "PHOSPHATE_FILTERED_AS P_UG/L"
#> [19] "AMMONIA_FILTERED_AS N_MG/L"
#> [20] "NITRATE + NITRITE_FILTERED_AS N_MG/L"
#> [21] "CHLOROPHYLL A_UNFILTERED_NA_UG/L"
#> [22] "CHLORIDE_TOTAL_NA_UG/L"
#> [23] "CALCIUM CARBONATE_TOTAL_NA_UG/L"
#> [24] "POTASSIUM_TOTAL_NA_UG/L"
#> [25] "SODIUM_TOTAL_NA_UG/L"
#> [26] "SULFATE_TOTAL_NA_UG/L"
#> [27] "MAGNESIUM_TOTAL_NA_UG/L"
#> [28] "CALCIUM_TOTAL_NA_UG/L"
#> [29] "STREAM STAGE_NA_NA_FT"
#> [30] "CONDUCTIVITY_NA_NA_US/CM"
#> [31] "PH_NA_NA_NA"
#> [32] "TRANSPARENCY, SECCHI TUBE WITH DISK_NA_NA_CM"
#> [33] "TRANSPARENCY, TUBE WITH DISK_TOTAL_NA_NTU"
#> [34] "VELOCITY - STREAM_NA_NA_M/SEC"
#> [35] "OXIDATION REDUCTION POTENTIAL (ORP)_NA_NA_VOLTS"
#> [36] "SALINITY_NA_NA_PSS"
#> [37] "FECAL COLIFORM_NA_NA_MPN/100ML"
#> [38] "TEMPERATURE, SAMPLE_NA_NA_DEG C"
#> [39] "CHLOROPHYLL A, CORRECTED FOR PHEOPHYTIN_TOTAL_NA_UG/L"
#> [40] "CHLOROPHYLL A, UNCORRECTED FOR PHEOPHYTIN_TOTAL_NA_UG/L"
#> [41] "CHLOROPHYLL B_TOTAL_NA_UG/L"
#> [42] "CHLOROPHYLL C_TOTAL_NA_UG/L"
#> [43] "PHEOPHYTIN A_TOTAL_NA_UG/L"
#> [44] "ORGANIC CARBON_TOTAL_NA_UG/L"
#> [45] "AMMONIA_UNFILTERED_AS N_MG/L"
#> [46] "PHOSPHORUS_DISSOLVED_NA_UG/L"
#> [47] "PHENOLS_TOTAL_NA_UG/L"
#> [48] "HARDNESS, CA, MG_NA_NA_UG/L"
#> [49] "LEAD_TOTAL_NA_UG/L"
#> [50] "MANGANESE_TOTAL_NA_UG/L"
#> [51] "NICKEL_TOTAL_NA_UG/L"
#> [52] "SELENIUM_TOTAL_NA_UG/L"
#> [53] "SILVER_TOTAL_NA_UG/L"
#> [54] "STRONTIUM_TOTAL_NA_UG/L"
#> [55] "VANADIUM_TOTAL_NA_UG/L"
#> [56] "ZINC_TOTAL_NA_UG/L"
#> [57] "ARSENIC_TOTAL_NA_UG/L"
#> [58] "BERYLLIUM_TOTAL_NA_UG/L"
#> [59] "BORON_TOTAL_NA_UG/L"
#> [60] "CADMIUM_TOTAL_NA_UG/L"
#> [61] "CHROMIUM_TOTAL_NA_UG/L"
#> [62] "COBALT_TOTAL_NA_UG/L"
#> [63] "COPPER_TOTAL_NA_UG/L"
#> [64] "ALUMINUM_DISSOLVED_NA_UG/L"
#> [65] "ARSENIC_DISSOLVED_NA_UG/L"
#> [66] "BARIUM_DISSOLVED_NA_UG/L"
#> [67] "BERYLLIUM_DISSOLVED_NA_UG/L"
#> [68] "BORON_DISSOLVED_NA_UG/L"
#> [69] "CADMIUM_DISSOLVED_NA_UG/L"
#> [70] "CALCIUM_DISSOLVED_NA_UG/L"
#> [71] "CHROMIUM_DISSOLVED_NA_UG/L"
#> [72] "COBALT_DISSOLVED_NA_UG/L"
#> [73] "COPPER_DISSOLVED_NA_UG/L"
#> [74] "IRON_DISSOLVED_NA_UG/L"
#> [75] "LEAD_DISSOLVED_NA_UG/L"
#> [76] "MAGNESIUM_DISSOLVED_NA_UG/L"
#> [77] "MANGANESE_DISSOLVED_NA_UG/L"
#> [78] "NICKEL_DISSOLVED_NA_UG/L"
#> [79] "POTASSIUM_DISSOLVED_NA_UG/L"
#> [80] "SELENIUM_DISSOLVED_NA_UG/L"
#> [81] "SILVER_DISSOLVED_NA_UG/L"
#> [82] "SODIUM_DISSOLVED_NA_UG/L"
#> [83] "STRONTIUM_DISSOLVED_NA_UG/L"
#> [84] "VANADIUM_DISSOLVED_NA_UG/L"
#> [85] "ZINC_DISSOLVED_NA_UG/L"
#> [86] "CYANIDE, AVAILABLE_NA_NA_UG/L"
#> [87] "NITRATE + NITRITE_UNFILTERED_AS N_MG/L"
#> [88] "FLUORIDE_TOTAL_NA_UG/L"
#> [89] "VOLATILE SUSPENDED SOLIDS_NA_NA_UG/L"
#> [90] "IRON_TOTAL_NA_UG/L"
#> [91] "ALUMINUM_TOTAL_NA_UG/L"
#> [92] "BARIUM_TOTAL_NA_UG/L"
#> [93] "CYANIDE_TOTAL_NA_UG/L"
#> [94] "GLYPHOSATE_TOTAL_NA_UG/L"
#> [95] "ACETOCHLOR_TOTAL_NA_UG/L"
#> [96] "ALACHLOR_TOTAL_NA_UG/L"
#> [97] "ALDRIN_TOTAL_NA_UG/L"
#> [98] ".ALPHA.-HEXACHLOROCYCLOHEXANE_TOTAL_NA_UG/L"
#> [99] "CIS-CHLORDANE_TOTAL_NA_UG/L"
#> [100] "CAPTAN_TOTAL_NA_UG/L"
#> [101] "CYANAZINE_TOTAL_NA_UG/L"
#> [102] "DIELDRIN_TOTAL_NA_UG/L"
#> [103] "ENDRIN_TOTAL_NA_UG/L"
#> [104] "LINDANE_TOTAL_NA_UG/L"
#> [105] "TRANS-CHLORDANE_TOTAL_NA_UG/L"
#> [106] "HEPTACHLOR_TOTAL_NA_UG/L"
#> [107] "HEPTACHLOR EPOXIDE_TOTAL_NA_UG/L"
#> [108] "HEXACHLOROBENZENE_TOTAL_NA_UG/L"
#> [109] "METHOXYCHLOR_TOTAL_NA_UG/L"
#> [110] "METOLACHLOR_TOTAL_NA_UG/L"
#> [111] "METRIBUZIN_TOTAL_NA_UG/L"
#> [112] "P,P'-DDD_TOTAL_NA_UG/L"
#> [113] "P,P'-DDE_TOTAL_NA_UG/L"
#> [114] "P,P'-DDT_TOTAL_NA_UG/L"
#> [115] "PENDIMETHALIN_TOTAL_NA_UG/L"
#> [116] "TOXAPHENE_TOTAL_NA_UG/L"
#> [117] "TRIFLURALIN_TOTAL_NA_UG/L"
#> [118] "ATRAZINE_TOTAL_NA_UG/L"
#> [119] "BUTYLATE_TOTAL_NA_UG/L"
#> [120] "CHLORPYRIFOS_TOTAL_NA_UG/L"
#> [121] "DIAZINON_TOTAL_NA_UG/L"
#> [122] "S-ETHYL DIPROPYLTHIOCARBAMATE_TOTAL_NA_UG/L"
#> [123] "PARATHION_TOTAL_NA_UG/L"
#> [124] "FONOFOS_TOTAL_NA_UG/L"
#> [125] "MALATHION_TOTAL_NA_UG/L"
#> [126] "METHYL PARATHION_TOTAL_NA_UG/L"
#> [127] "PHORATE_TOTAL_NA_UG/L"
#> [128] "SIMAZINE_TOTAL_NA_UG/L"
#> [129] "TERBUFOS_TOTAL_NA_UG/L"
#> [130] "POLYCHLORINATED BIPHENYLS_TOTAL_NA_UG/L"
#> [131] "2,4-D_TOTAL_NA_UG/L"
#> [132] "ACIFLUORFEN_TOTAL_NA_UG/L"
#> [133] "DALAPON_TOTAL_NA_UG/L"
#> [134] "DICAMBA_TOTAL_NA_UG/L"
#> [135] "DINOSEB_TOTAL_NA_UG/L"
#> [136] "PENTACHLOROPHENOL_TOTAL_NA_UG/L"
#> [137] "PICLORAM_TOTAL_NA_UG/L"
#> [138] "SILVEX_TOTAL_NA_UG/L"
#> [139] "TOTAL DISSOLVED SOLIDS_NA_NA_UG/L"
#> [140] "TOTAL DISSOLVED SOLIDS_TOTAL_NA_UG/L"
#> [141] "FLOW_TOTAL_NA_CFS"
#> [142] "SPECIFIC CONDUCTANCE_TOTAL_NA_US/CM"
#> [143] "OXIDATION REDUCTION POTENTIAL (ORP)_TOTAL_NA_VOLTS"
#> [144] "TURBIDITY_TOTAL_NA_NTU"
#> [145] "TOTAL SUSPENDED SOLIDS_FILTERED, LAB_NA_UG/L"
#> [146] "ALUMINUM_TOTAL RECOVERABLE_NA_UG/L"
#> [147] "BARIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [148] "BERYLLIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [149] "CADMIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [150] "CALCIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [151] "CHROMIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [152] "COBALT_TOTAL RECOVERABLE_NA_UG/L"
#> [153] "ESCHERICHIA COLI_NA_NA_MPN"
#> [154] "COPPER_TOTAL RECOVERABLE_NA_UG/L"
#> [155] "IRON_TOTAL RECOVERABLE_NA_UG/L"
#> [156] "LEAD_TOTAL RECOVERABLE_NA_UG/L"
#> [157] "MAGNESIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [158] "MANGANESE_TOTAL RECOVERABLE_NA_UG/L"
#> [159] "MERCURY_DISSOLVED_NA_UG/L"
#> [160] "MERCURY_TOTAL RECOVERABLE_NA_UG/L"
#> [161] "MOLYBDENUM_DISSOLVED_NA_UG/L"
#> [162] "MOLYBDENUM_TOTAL RECOVERABLE_NA_UG/L"
#> [163] "NICKEL_TOTAL RECOVERABLE_NA_UG/L"
#> [164] "TOTAL NITROGEN, MIXED FORMS_UNFILTERED_AS N_MG/L"
#> [165] "POTASSIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [166] "SILVER_TOTAL RECOVERABLE_NA_UG/L"
#> [167] "SODIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [168] "SULFATE_TOTAL_AS SO4_UG/L"
#> [169] "THALLIUM_DISSOLVED_NA_UG/L"
#> [170] "THALLIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [171] "TITANIUM_DISSOLVED_NA_UG/L"
#> [172] "TITANIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [173] "ORGANIC CARBON_DISSOLVED_NA_UG/L"
#> [174] "VANADIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [175] "ZINC_TOTAL RECOVERABLE_NA_UG/L"
#> [176] "HEIGHT, GAGE_NA_NA_FT"
#> [177] "BIOCHEMICAL OXYGEN DEMAND, STANDARD CONDITIONS_UNFILTERED_NA_UG/L"
#> [178] "CHLORIDE_UNFILTERED_NA_UG/L"
#> [179] "CHLOROPHYLL A_NON-FILTERABLE (PARTICLE)_NA_UG/L"
#> [180] "FECAL COLIFORM_UNFILTERED_NA_#/100ML"
#> [181] "INORGANIC CARBON_UNFILTERED_AS C_UG/L"
#> [182] "NITRATE_UNFILTERED_AS N_MG/L"
#> [183] "NITRITE_UNFILTERED_AS N_MG/L"
#> [184] "ORGANIC CARBON_UNFILTERED_AS C_UG/L"
#> [185] "ORGANIC CARBON_FILTERED, LAB_AS C_UG/L"
#> [186] "TOTAL PHOSPHORUS, MIXED FORMS_FILTERED_AS P_UG/L"
#> [187] "VOLATILE SUSPENDED SOLIDS_VOLATILE_NA_UG/L"
#> [188] "TOTAL SOLIDS_UNFILTERED_NA_UG/L"
#> [189] "ARSENIC_TOTAL RECOVERABLE_NA_UG/L"
#> [190] "SELENIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [191] "TOTAL HARDNESS_TOTAL RECOVERABLE_NA_UG/L"
#> [192] "MAGNESIUM_TOTAL_AS MG_UG/L"
#> [193] "CALCIUM_TOTAL_AS CA_UG/L"
#> [194] "SULFATE_UNFILTERED_AS SO4_UG/L"
#> [195] "CHEMICAL OXYGEN DEMAND_NA_NA_UG/L"
#> [196] "CARBONACEOUS BIOCHEMICAL OXYGEN DEMAND, STANDARD CONDITIONS_NA_NA_UG/L"
#> [197] "STRONTIUM_TOTAL RECOVERABLE_NA_UG/L"
#> [198] "TOTAL COLIFORM_TOTAL_NA_MPN/100ML"
#> [199] "TEMPERATURE, AIR_NA_NA_DEG C"
#> [200] "STREAM STAGE_TOTAL_NA_FT"
#> [201] "VOLATILE SUSPENDED SOLIDS_TOTAL_NA_UG/L"
#> [202] "SALINITY_TOTAL_NA_UG/KG"
#> [203] "HEIGHT, GAGE_TOTAL_NA_FT"
#> [204] "DISTANCE FROM/TO_TOTAL_NA_FT"
#> [205] "HARDNESS, CARBONATE_TOTAL_AS CACO3_UG/L"
#> [206] "TEMPERATURE, AIR_TOTAL_NA_DEG C"
#> [207] "APPARENT COLOR_TOTAL_NA_PCU"
#> [208] "TRUE COLOR_TOTAL_NA_PCU"
#> [209] "TOTAL VOLATILE SOLIDS_TOTAL_NA_UG/L"
#> [210] "CHEMICAL OXYGEN DEMAND_TOTAL_NA_UG/L"
#> [211] "BIOCHEMICAL OXYGEN DEMAND, STANDARD CONDITIONS_TOTAL_NA_UG/L"
#> [212] "CARBONACEOUS BIOCHEMICAL OXYGEN DEMAND, STANDARD CONDITIONS_TOTAL_NA_UG/L"
#> [213] "FECAL COLIFORM_NA_NA_#/100ML"
#> [214] "TOTAL PHOSPHORUS, MIXED FORMS_TOTAL_AS P_UG/L"
#> [215] "BORON_TOTAL_AS B_UG/L"
#> [216] "ARSENIC_TOTAL_AS AS_UG/L"
#> [217] "CHROMIUM_TOTAL_AS CR_UG/L"
#> [218] "NICKEL_TOTAL_AS NI_UG/L"
#> [219] "SELENIUM_TOTAL_AS SE_UG/L"
#> [220] "TOTAL SOLIDS_TOTAL_NA_UG/L"
#> [221] "TOTAL DISSOLVED SOLIDS_DISSOLVED_NA_UG/L"
#> [222] "IRON_TOTAL_AS FE_UG/L"
#> [223] "BENZO[A]PYRENE_TOTAL_NA_UG/L"
#> [224] "DI(2-ETHYLHEXYL) ADIPATE_TOTAL_NA_UG/L"
#> [225] "DI(2-ETHYLHEXYL) PHTHALATE_TOTAL_NA_UG/L"
#> [226] "2-CHORO-6-ETHYLAMINO-4-AMINO-S-TRIAZINE_TOTAL_NA_UG/L"
#> [227] "2-CHLORO-4-ISOPROPYLAMINO-6-AMINO-S-TRIAZINE_TOTAL_NA_UG/L"
#> [228] "OXYCHLORDANE_TOTAL_NA_UG/L"
#> [229] "CLOMAZONE_TOTAL_NA_UG/L"
#> [230] "CIS-NONACHLOR_TOTAL_NA_UG/L"
#> [231] "PROPACHLOR_TOTAL_NA_UG/L"
#> [232] "HEXACHLOROCYCLOPENTADIENE_TOTAL_NA_UG/L"
#> [233] "TRANS-NONACHLOR_TOTAL_NA_UG/L"
#> [234] "FLUORIDE_TOTAL_AS F_UG/L"
#> [235] "METRIBUZIN DA_TOTAL_NA_UG/L"
#> [236] "DEETHYLCYANAZINE AMIDE_TOTAL_NA_UG/L"
#> [237] "CYANAZINE AMIDE_TOTAL_NA_UG/L"
#> [238] "CYANAZINE ACID_TOTAL_NA_UG/L"
#> [239] "DEETHYLCYANAZINE ACID_TOTAL_NA_UG/L"
#> [240] "HEXAZINONE_TOTAL_NA_UG/L"
#> [241] "METRIBUZIN DADK_TOTAL_NA_UG/L"
#> [242] "TRICLOPYR_TOTAL_NA_UG/L"
#> [243] "METRIBUZIN DK_TOTAL_NA_UG/L"
#> [244] "METALAXYL_TOTAL_NA_UG/L"
#> [245] "PROPICONAZOLE_TOTAL_NA_UG/L"
#> [246] "PROPACHLOR OA_TOTAL_NA_UG/L"
#> [247] "FOMESAFEN_TOTAL_NA_UG/L"
#> [248] "SULFOMETURON METHYL_TOTAL_NA_UG/L"
#> [249] "METSULFURON-METHYL_TOTAL_NA_UG/L"
#> [250] "FLUTRIAFOL_TOTAL_NA_UG/L"
#> [251] "THIFENSULFURON-METHYL_TOTAL_NA_UG/L"
#> [252] "IMAZAPYR_TOTAL_NA_UG/L"
#> [253] "IMAZAQUIN_TOTAL_NA_UG/L"
#> [254] "IMAZETHAPYR_TOTAL_NA_UG/L"
#> [255] "IMAZAMETHABENZ-METHYL_TOTAL_NA_UG/L"
#> [256] "TRIASULFURON_TOTAL_NA_UG/L"
#> [257] "BENSULFURON-METHYL_TOTAL_NA_UG/L"
#> [258] "MYCLOBUTANIL_TOTAL_NA_UG/L"
#> [259] "CHLORIMURON-ETHYL_TOTAL_NA_UG/L"
#> [260] "FLUMETSULAM_TOTAL_NA_UG/L"
#> [261] "TOLFENPYRAD_TOTAL_NA_UG/L"
#> [262] "IMAZAMETHABENZ ACID_TOTAL_NA_UG/L"
#> [263] "HALOSULFURON-METHYL_TOTAL_NA_UG/L"
#> [264] "IMAZAPIC_TOTAL_NA_UG/L"
#> [265] "MESOTRIONE_TOTAL_NA_UG/L"
#> [266] "TEBUCONAZOLE_TOTAL_NA_UG/L"
#> [267] "CLETHODIM SULFOXIDE_TOTAL_NA_UG/L"
#> [268] "CLETHODIM SULFONE_TOTAL_NA_UG/L"
#> [269] "THIACLOPRID_TOTAL_NA_UG/L"
#> [270] "DICROTOPHOS_TOTAL_NA_UG/L"
#> [271] "OXYDEMETON-METHYL_TOTAL_NA_UG/L"
#> [272] "BROMACIL_TOTAL_NA_UG/L"
#> [273] "DIURON_TOTAL_NA_UG/L"
#> [274] "LINURON_TOTAL_NA_UG/L"
#> [275] "BENSULIDE_TOTAL_NA_UG/L"
#> [276] "METHYL PARAOXON_TOTAL_NA_UG/L"
#> [277] "DICHLOBENIL_TOTAL_NA_UG/L"
#> [278] "PROMETON_TOTAL_NA_UG/L"
#> [279] "BENFLURALIN_TOTAL_NA_UG/L"
#> [280] "CHLOROTHALONIL_TOTAL_NA_UG/L"
#> [281] "TRIALLATE_TOTAL_NA_UG/L"
#> [282] "DICHLORVOS_TOTAL_NA_UG/L"
#> [283] "DIMETHOATE_TOTAL_NA_UG/L"
#> [284] "CARBOFURAN_TOTAL_NA_UG/L"
#> [285] "ALDICARB SULFOXIDE_TOTAL_NA_UG/L"
#> [286] "ALDICARB SULFONE_TOTAL_NA_UG/L"
#> [287] "BROMOXYNIL_TOTAL_NA_UG/L"
#> [288] "CLOPYRALID_TOTAL_NA_UG/L"
#> [289] "SIDURON_TOTAL_NA_UG/L"
#> [290] "2-HYDROXYATRAZINE_TOTAL_NA_UG/L"
#> [291] "DISULFOTON SULFONE_TOTAL_NA_UG/L"
#> [292] "2-CHLORO-4,6-DIAMINO-S-TRIAZINE_TOTAL_NA_UG/L"
#> [293] "CHLORPYRIFOS O.A._TOTAL_NA_UG/L"
#> [294] "PROMETRYN_TOTAL_NA_UG/L"
#> [295] "OXADIAZON_TOTAL_NA_UG/L"
#> [296] "ETHOFUMESATE_TOTAL_NA_UG/L"
#> [297] "CARBARYL_TOTAL_NA_UG/L"
#> [298] "MECOPROP_TOTAL_NA_UG/L"
#> [299] "2,4,5-T_TOTAL_NA_UG/L"
#> [300] "MCPA_TOTAL_NA_UG/L"
#> [301] "MCPB_TOTAL_NA_UG/L"
#> [302] "2,4-DB_TOTAL_NA_UG/L"
#> [303] "PROPAZINE_TOTAL_NA_UG/L"
#> [304] "DISULFOTON_TOTAL_NA_UG/L"
#> [305] "DIAZOXON_TOTAL_NA_UG/L"
#> [306] "DICHLORPROP_TOTAL_NA_UG/L"
#> [307] "CYANTRANILIPROLE_TOTAL_NA_UG/L"
#> [308] "SEDAXANE_TOTAL_NA_UG/L"
#> [309] "FLUXAPYROXAD_TOTAL_NA_UG/L"
#> [310] "AFIDOPYROPEN_TOTAL_NA_UG/L"
#> [311] "HALAUXIFEN-METHYL_TOTAL_NA_UG/L"
#> [312] "HALAUXIFEN ACID_TOTAL_NA_UG/L"
#> [313] "PROPACHLOR ESA_TOTAL_NA_UG/L"
#> [314] "FLUPYRADIFURONE_TOTAL_NA_UG/L"
#> [315] "FLUTIANIL_TOTAL_NA_UG/L"
#> [316] "OXATHIAPIPROLIN_TOTAL_NA_UG/L"
#> [317] "BENZOVINDIFLUPYR_TOTAL_NA_UG/L"
#> [318] "TOLPYRALATE_TOTAL_NA_UG/L"
#> [319] "PYDIFLUMETOFEN_TOTAL_NA_UG/L"
#> [320] "FLUTIANIL OC 56574_TOTAL_NA_UG/L"
#> [321] "FLUTIANIL OC 56635_TOTAL_NA_UG/L"
#> [322] "BETA CYPERMETHRIN_TOTAL_NA_UG/L"
#> [323] "ETHALFLURALIN_TOTAL_NA_UG/L"
#> [324] "ESFENVALERATE_TOTAL_NA_UG/L"
#> [325] "CYFLUTHRIN_TOTAL_NA_UG/L"
#> [326] "BIFENTHRIN_TOTAL_NA_UG/L"
#> [327] "DIMETHENAMID_TOTAL_NA_UG/L"
#> [328] ".LAMBDA.-CYHALOTHRIN_TOTAL_NA_UG/L"
#> [329] "PHOSTEBUPIRIM_TOTAL_NA_UG/L"
#> [330] "CARBENDAZIM_TOTAL_NA_UG/L"
#> [331] "DEETHYLCYANAZINE_TOTAL_NA_UG/L"
#> [332] "DESMETHYLNORFLURAZON_TOTAL_NA_UG/L"
#> [333] "BENTAZON_TOTAL_NA_UG/L"
#> [334] "NORFLURAZON_TOTAL_NA_UG/L"
#> [335] "THIOBENCARB_TOTAL_NA_UG/L"
#> [336] "METOLACHLOR ESA_TOTAL_NA_UG/L"
#> [337] "ALACHLOR OA_TOTAL_NA_UG/L"
#> [338] "MANDESTROBIN_TOTAL_NA_UG/L"
#> [339] "PYRACLOSTROBIN_TOTAL_NA_UG/L"
#> [340] "ACETOCHLOR ESA_TOTAL_NA_UG/L"
#> [341] "BOSCALID_TOTAL_NA_UG/L"
#> [342] "ACETOCHLOR OA_TOTAL_NA_UG/L"
#> [343] "FLUFENACET OA_TOTAL_NA_UG/L"
#> [344] "DIMETHENAMID ESA_TOTAL_NA_UG/L"
#> [345] "CLOTHIANIDIN_TOTAL_NA_UG/L"
#> [346] "TEMBOTRIONE_TOTAL_NA_UG/L"
#> [347] "BICYCLOPYRONE_TOTAL_NA_UG/L"
#> [348] "SAFLUFENACIL_TOTAL_NA_UG/L"
#> [349] "BICYCLOPYRONE SYN503870_TOTAL_NA_UG/L"
#> [350] "DIMETHENAMID OA_TOTAL_NA_UG/L"
#> [351] "PYROXASULFONE_TOTAL_NA_UG/L"
#> [352] "CHLORANTRANILIPROLE_TOTAL_NA_UG/L"
#> [353] "MOMFLUOROTHRIN_TOTAL_NA_UG/L"
#> [354] "NICOSULFURON_TOTAL_NA_UG/L"
#> [355] "TETRACONAZOLE_TOTAL_NA_UG/L"
#> [356] "IMAZAMOX_TOTAL_NA_UG/L"
#> [357] "IMIDACLOPRID-OLEFIN_TOTAL_NA_UG/L"
#> [358] "PICOXYSTROBIN_TOTAL_NA_UG/L"
#> [359] "DIFENOCONAZOLE_TOTAL_NA_UG/L"
#> [360] "IMIDACLOPRID-UREA_TOTAL_NA_UG/L"
#> [361] "SULFENTRAZONE_TOTAL_NA_UG/L"
#> [362] "AZOXYSTROBIN_TOTAL_NA_UG/L"
#> [363] "ACETAMIPRID_TOTAL_NA_UG/L"
#> [364] "IMIDACLOPRID_TOTAL_NA_UG/L"
#> [365] "ISOXAFLUTOLE_TOTAL_NA_UG/L"
#> [366] "ALACHLOR ESA_TOTAL_NA_UG/L"
#> [367] "BENZENEPROPANENITRILE, .ALPHA.-(CYCLOPROPYLCARBONYL)- 2-(METHYLSULFONYL)-.BETA.- OXO-4-(TRIFLUOROMETHYL)-_TOTAL_NA_UG/L"
#> [368] "AMINOPYRALID_TOTAL_NA_UG/L"
#> [369] "METOLACHLOR OA_TOTAL_NA_UG/L"
#> [370] "THIAMETHOXAM_TOTAL_NA_UG/L"
#> [371] "DINOTEFURAN_TOTAL_NA_UG/L"
#> [372] "TRANSPARENCY, TUBE WITH DISK_NA_NA_CM"
#> [373] "DEPTH, BOTTOM_NA_NA_M"
#> [374] "CHLOROPHYLL A, FREE OF PHEOPHYTIN_NON-FILTERABLE (PARTICLE)_NA_UG/L"
#> [375] "AMMONIUM_FILTERABLE_AS N_UG/L"
#> [376] "VELOCITY-DISCHARGE_NA_NA_CFS"
#> [377] "TOTAL NITROGEN, MIXED FORMS_UNFILTERED_AS N_UG/L"
# choose two and generate scatterplot
DO_Temp_Scatter <- TADA_TwoCharacteristicScatterplot(R5ProfileClean8, id_cols = "TADA.ComparableDataIdentifier", groups = c("TEMPERATURE_NA_NA_DEG C", "DISSOLVED OXYGEN (DO)_NA_NA_MG/L"))
DO_Temp_ScatterNow we will summarize results for a single comparable data group using the TADA.ComparableDataIdentifier (i.e., comparable characteristic, unit, speciation, and fraction combination) using TADA_Histogram and TADA_Boxplot. Note that users may generate a list output of multiple plots if their input dataset has more than one unique comparable data group.
# filter dataframe to only "DISSOLVED OXYGEN (DO)_NA_NA_MG/L"
R5ProfileCleanDO <- dplyr::filter(R5ProfileClean8, TADA.ComparableDataIdentifier == "DISSOLVED OXYGEN (DO)_NA_NA_MG/L")
# generate a histogram
DO_Hist <- TADA_Histogram(R5ProfileCleanDO, id_cols = "TADA.ComparableDataIdentifier")
DO_Hist
# generate stats table
R5ProfileCleanDO_stats <- TADA_Stats(R5ProfileCleanDO)
#> [1] "Note: Your dataset contains TADA-generated total nutrient results, which have fewer columns populated with metadata. This might affect how groups are displayed in the stats table."Generate interactive box plot.
DO_Box <- TADA_Boxplot(R5ProfileCleanDO, id_cols = "TADA.ComparableDataIdentifier")
DO_BoxGenerate interactive scatterplot.
R5ProfileCleanDO_dailymax <- TADA_AggregateMeasurements(R5ProfileCleanDO,
agg_fun = c("max"),
clean = TRUE
)
#> [1] "Aggregation results:"
#>
#> No aggregation needed Selected as max aggregate value
#> 383 12
DO_Scatter <- TADA_Scatterplot(R5ProfileCleanDO_dailymax, id_cols = "TADA.ComparableDataIdentifier")
DO_ScatterQuestion 15: Which figure generating functions can be used to visualize more than one characteristic? Which can be used to visualize only one characteristic?
Answer: TADA_TwoCharacteristicScatterplot can display up to two characteristics. TADA_Boxplot, TADA_Histogram, and TADA_Scatterplot will show only one characteristic per plot, but can be used to generate a a list output of multiple plots (if input contains more than one comparable data group).
Retain TADA Required Columns
Now we can review the “TADA” prefixed columns we have added to the data set. If we are satisfied with the conversions, filtering, flagging, etc. and the resulting “TADA” columns, we can use the TADA_RetainRequired function to remove any columns that are not required or used as filters in the TADA workflow. This reduces the size of the data frame.
R5ProfileClean9 <- TADA_RetainRequired(R5ProfileClean8)
#> [1] "TADA_RetainRequired: removing columns not required for TADA workflow including original columns that have been replaced with TADA prefix duplicates."
#> [1] "The following non-required columns were removed: ActivityEndDate, ActivityEndTime.Time, ActivityEndTime.TimeZoneCode, ActivityEndDateTime, ActivityConductingOrganizationText, ActivityLocation.LatitudeMeasure, ActivityLocation.LongitudeMeasure, ResultStatusIdentifier, ResultWeightBasisText, ResultTemperatureBasisText, AnalysisStartDate, ResultDetectionQuantitationLimitUrl, timeZoneStart, timeZoneEnd, SourceMapScaleNumeric, HorizontalCollectionMethodName, HorizontalCoordinateReferenceSystemDatumName, VerticalMeasure.MeasureValue, VerticalMeasure.MeasureUnitCode, VerticalCollectionMethodName and VerticalCoordinateReferenceSystemDatumName."Question 16: How many columns were removed during TADA_RetainRequired?
R5ProfileClean8_n <- ncol(R5ProfileClean8)
R5ProfileClean9_n <- ncol(R5ProfileClean9)
retain_required_n <- R5ProfileClean8_n - R5ProfileClean9_nAnswer:
19 columns were removed by TADA_RetainRequired.
TADA Shiny Application
Finally, take a look at an alternative workflow, TADA Shiny Module 1: Data Discovery and Cleaning. This is a Shiny application that runs many of the TADA functions covered in this document behind a graphical user interface. The shiny application queries the WQP, contains maps and data visualizations, flags suspect data results, handles censored data, and more. You can launch it using the code below.
DRAFT Module 1 is also currently hosted on the web with minimal server memory/storage allocated.
# download TADA Shiny repository
remotes::install_github("USEPA/TADAShiny",
ref = "develop",
dependencies = TRUE
)
# launch the app locally.
TADAShiny::run_app()