Title: | Support for Parallel Computation, Logging, and Function Automation |
---|---|
Description: | Support for parallel computation with progress bar, and option to stop or proceed on errors. Also provides logging to console and disk, and the logging persists in the parallel threads. Additional functions support function call automation with delayed execution (e.g. for executing functions in parallel). |
Authors: | Martijn Schuemie [aut, cre], Marc Suchard [aut], Observational Health Data Science and Informatics [cph] |
Maintainer: | Martijn Schuemie <[email protected]> |
License: | Apache License 2.0 |
Version: | 3.3.1 |
Built: | 2024-11-03 05:06:30 UTC |
Source: | https://github.com/ohdsi/parallellogger |
Add the default console logger
addDefaultConsoleLogger(name = "DEFAULT_CONSOLE_LOGGER")
addDefaultConsoleLogger(name = "DEFAULT_CONSOLE_LOGGER")
name |
A name for the logger. |
Creates a logger that writes to the console using the "INFO" threshold and the
layoutSimple
layout.
logger <- addDefaultConsoleLogger() logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger(logger)
logger <- addDefaultConsoleLogger() logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger(logger)
Add the default e-mail logger
addDefaultEmailLogger( mailSettings, label = Sys.info()["nodename"], name = "DEFAULT_EMAIL_LOGGER", test = FALSE )
addDefaultEmailLogger( mailSettings, label = Sys.info()["nodename"], name = "DEFAULT_EMAIL_LOGGER", test = FALSE )
mailSettings |
Arguments to be passed to the |
label |
A label to be used in the e-mail subject to identify a run. By default the name of the computer is used. |
name |
A name for the logger. |
test |
If TRUE, a message will be displayed on the console instead of sending an e-mail. |
Creates a logger that writes to e-mail using the "FATAL" threshold and the
layoutEmail
layout. This function uses the sendmailR
package. Please make sure
your e-mail settings are correct by using the sendmailR
package before using those settings here.
ParallelLogger
will not display any messages if something goes wrong when sending the e-mail.
To use a GMail account, make sure to enable 2-step verification on your
Google account (see 'Security'). Click on 2-Step Verification, and
scroll down to 'App passwords'. Here, you can create an app-specific password
to be used with ParallelLogger
. You can set
host.name = "smtp.gmail.com:587"
, and be sure to use
engine = "curl"
.
mailSettings <- list( from = "[email protected]", to = "[email protected]", engine = "curl", engineopts = list( username = "[email protected]", password = "Secret!" ), control = list( host.name = "smtp.gmail.com:587" ) ) # Setting test to TRUE in this example so we don't really send an e-mail: addDefaultEmailLogger(mailSettings, "My R session", test = TRUE) logFatal("Something bad") unregisterLogger("DEFAULT_EMAIL_LOGGER")
mailSettings <- list( from = "[email protected]", to = "[email protected]", engine = "curl", engineopts = list( username = "[email protected]", password = "Secret!" ), control = list( host.name = "smtp.gmail.com:587" ) ) # Setting test to TRUE in this example so we don't really send an e-mail: addDefaultEmailLogger(mailSettings, "My R session", test = TRUE) logFatal("Something bad") unregisterLogger("DEFAULT_EMAIL_LOGGER")
Add the default error report logger
addDefaultErrorReportLogger( fileName = file.path(getwd(), "errorReportR.txt"), name = "DEFAULT_ERRORREPORT_LOGGER" )
addDefaultErrorReportLogger( fileName = file.path(getwd(), "errorReportR.txt"), name = "DEFAULT_ERRORREPORT_LOGGER" )
fileName |
The name of the file to write to. |
name |
A name for the logger. |
Creates a logger that writes to a file using the "FATAL" threshold and the
layoutErrorReport
layout. The file will be overwritten if it is older than 60
seconds. The user will be notified that the error report has been created, and where to find it.
Add the default file logger
addDefaultFileLogger(fileName, name = "DEFAULT_FILE_LOGGER")
addDefaultFileLogger(fileName, name = "DEFAULT_FILE_LOGGER")
fileName |
The name of the file to write to. |
name |
A name for the logger. |
Creates a logger that writes to a file using the "TRACE" threshold and the
layoutParallel
layout. The output can be viewed with the built-in log viewer that can
be started using launchLogViewer
.
Remove all registered loggers
clearLoggers()
clearLoggers()
Apply a function to a list using the cluster
clusterApply(cluster, x, fun, ..., stopOnError = FALSE, progressBar = TRUE)
clusterApply(cluster, x, fun, ..., stopOnError = FALSE, progressBar = TRUE)
cluster |
The cluster of threads to run the function. |
x |
The list on which the function will be applied. |
fun |
The function to apply. Note that the context in which the function is specifies matters (see details). |
... |
Additional parameters for the function. |
stopOnError |
Stop when one of the threads reports an error? If FALSE, all errors will be reported at the end. |
progressBar |
Show a progress bar? |
The function will be executed on each element of x in the threads of the cluster. If there are more elements than threads, the elements will be queued. The progress bar will show the number of elements that have been completed. It can sometimes be important to realize that the context in which a function is created is also transmitted to the worker node. If a function is defined inside another function, and that outer function is called with a large argument, that argument will be transmitted to the worker node each time the function is executed. It can therefore make sense to define the function to be called at the package level rather than inside a function, to save overhead.
A list with the result of the function on each item in x.
fun <- function(x) { return (x^2) } cluster <- makeCluster(numberOfThreads = 3) clusterApply(cluster, 1:10, fun) stopCluster(cluster)
fun <- function(x) { return (x^2) } cluster <- makeCluster(numberOfThreads = 3) clusterApply(cluster, 1:10, fun) stopCluster(cluster)
Calls the require
function in each node of the cluster.
clusterRequire(cluster, package)
clusterRequire(cluster, package)
cluster |
The cluster object. |
package |
The name of the package to load in all nodes. |
Converts a JSON string to a settings object
convertJsonToSettings(json)
convertJsonToSettings(json)
json |
A JSON string. |
Converts a JSON string generated using the convertSettingsToJson
function to a
settings object, restoring object classes and attributes.
An R object as specified by the JSON.
Convert a settings object to a JSON string
convertSettingsToJson(object)
convertSettingsToJson(object)
object |
R object to be converted. |
Convert a settings object to a JSON string, using pretty formatting and preserving object classes and attributes.
A JSON string representing the R object.
Create an argument function
createArgFunction( functionName, excludeArgs = c(), includeArgs = NULL, addArgs = list(), rCode = c(), newName )
createArgFunction( functionName, excludeArgs = c(), includeArgs = NULL, addArgs = list(), rCode = c(), newName )
functionName |
The name of the function for which we want to create an args function. |
excludeArgs |
Exclude these arguments from appearing in the args function. |
includeArgs |
Include these arguments in the args function. |
addArgs |
Add these arguments to the args functions. Defined as a list with format name = default. |
rCode |
A character vector representing the R code where the new function should be appended to. |
newName |
The name of the new function. If not specified, the new name will be automatically derived from the old name. |
This function can be used to create a function that has (almost) the same interface as the specified function, and the output of this function will be a list of argument values.
A character vector with the R code including the new function.
createArgFunction("read.csv", addArgs = list(exposureId = "exposureId"))
createArgFunction("read.csv", addArgs = list(exposureId = "exposureId"))
Create console appender
createConsoleAppender(layout = layoutSimple)
createConsoleAppender(layout = layoutSimple)
layout |
The layout to be used by the appender. |
Creates an appender that will write to the console.
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
Create e-mail appender
createEmailAppender( layout = layoutEmail, mailSettings, label = Sys.info()["nodename"], test = FALSE )
createEmailAppender( layout = layoutEmail, mailSettings, label = Sys.info()["nodename"], test = FALSE )
layout |
The layout to be used by the appender. |
mailSettings |
Arguments to be passed to the |
label |
A label to be used in the e-mail subject to identify a run. By default the name of the computer is used. |
test |
If TRUE, a message will be displayed on the console instead of sending an e-mail. |
Creates an appender that will send log events to an e-mail address using the sendmailR
package.
Please make sure your settings are correct by using the sendmailR
package before using those settings
here. ParallelLogger
will not display any messages if something goes wrong when sending the e-mail.
To use a GMail account, make sure to enable 2-step verification on your
Google account (see 'Security'). Click on 2-Step Verification, and
scroll down to 'App passwords'. Here, you can create an app-specific password
to be used with ParallelLogger
. You can set
host.name = "smtp.gmail.com:587"
, and be sure to use
engine = "curl"
.
mailSettings <- list( from = "[email protected]", to = "[email protected]", engine = "curl", engineopts = list( username = "[email protected]", password = "Secret!" ), control = list( host.name = "smtp.gmail.com:587" ) ) # Setting test to TRUE in this example so we don't really send an e-mail: appender <- createEmailAppender( layout = layoutEmail, mailSettings = mailSettings, label = "My R session", test = TRUE ) logger <- createLogger(name = "EMAIL", threshold = "FATAL", appenders = list(appender)) registerLogger(logger) logFatal("Something bad") unregisterLogger("EMAIL")
mailSettings <- list( from = "[email protected]", to = "[email protected]", engine = "curl", engineopts = list( username = "[email protected]", password = "Secret!" ), control = list( host.name = "smtp.gmail.com:587" ) ) # Setting test to TRUE in this example so we don't really send an e-mail: appender <- createEmailAppender( layout = layoutEmail, mailSettings = mailSettings, label = "My R session", test = TRUE ) logger <- createLogger(name = "EMAIL", threshold = "FATAL", appenders = list(appender)) registerLogger(logger) logFatal("Something bad") unregisterLogger("EMAIL")
Create file appender
createFileAppender( layout = layoutParallel, fileName, overwrite = FALSE, expirationTime = 60 )
createFileAppender( layout = layoutParallel, fileName, overwrite = FALSE, expirationTime = 60 )
layout |
The layout to be used by the appender. |
fileName |
The name of the file to write to. |
overwrite |
Overwrite the file if it is older than the expiration time? |
expirationTime |
Expiration time in seconds |
Creates an appender that will write to a file.
Create a logger
createLogger( name = "SIMPLE", threshold = "INFO", appenders = list(createConsoleAppender()) )
createLogger( name = "SIMPLE", threshold = "INFO", appenders = list(createConsoleAppender()) )
name |
A name for the logger. |
threshold |
The threshold to be used for reporting. |
appenders |
A list of one or more appenders as created for example using the
|
Creates a logger that will log messages to its appenders. The logger will only log messages at a level equal to or higher than its threshold. For example, if the threshold is "INFO" then messages marked "INFO" will be logged, but messages marked "TRACE" will not. The order of levels is "TRACE", "DEBUG", "INFO", "WARN", "ERROR, "and FATAL".
An object of type Logger
, to be used with the registerLogger
function.
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
Exclude variables from a list of objects of the same type
excludeFromList(x, exclude)
excludeFromList(x, exclude)
x |
A list of objects of the same type. |
exclude |
A character vector of names of variables to exclude. |
Get all registered loggers
getLoggers()
getLoggers()
Returns all registered loggers.
Launch the log viewer Shiny app
launchLogViewer(logFileName)
launchLogViewer(logFileName)
logFileName |
Name of the log file to view. |
Launches a Shiny app that allows the user to view a log file created using the default file logger.
Use addDefaultFileLogger
to start the default file logger.
# Create a log file: logFile <- file.path(tempdir(), "log.txt") addDefaultFileLogger(logFile) logInfo("Hello world") # Launch the log file viewer (only if in interactive mode): if (interactive()) { launchLogViewer(logFile) } # Delete the log file: unlink(logFile)
# Create a log file: logFile <- file.path(tempdir(), "log.txt") addDefaultFileLogger(logFile) logInfo("Hello world") # Launch the log file viewer (only if in interactive mode): if (interactive()) { launchLogViewer(logFile) } # Delete the log file: unlink(logFile)
A layout function to be used with an e-mail appender. This layout creates a short summary e-mail message on the event, including stack trace.
layoutEmail(level, message)
layoutEmail(level, message)
level |
The level of the message (e.g. "INFO") |
message |
The message to layout. |
A layout function to be used with an appender. This layout creates a more elaborate error message, for sharing with the developer. If an error occurs in the main thread a summary of the system info will be included.
layoutErrorReport(level, message)
layoutErrorReport(level, message)
level |
The level of the message (e.g. "INFO") |
message |
The message to layout. |
A layout function to be used with an appender. This layout adds the time, thread, level, package name, and function name to the message.
layoutParallel(level, message)
layoutParallel(level, message)
level |
The level of the message (e.g. "INFO") |
message |
The message to layout. |
A layout function to be used with an appender. This layout simply includes the message itself.
layoutSimple(level, message)
layoutSimple(level, message)
level |
The level of the message (e.g. "INFO") |
message |
The message to layout. |
A layout function to be used with an appender. This layout adds the stack trace to the message.
layoutStackTrace(level, message)
layoutStackTrace(level, message)
level |
The level of the message (e.g. "INFO") |
message |
The message to layout. |
A layout function to be used with an appender. This layout adds the time to the message.
layoutTimestamp(level, message)
layoutTimestamp(level, message)
level |
The level of the message (e.g. "INFO") |
message |
The message to layout. |
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
Load a settings object from a JSON file
loadSettingsFromJson(fileName)
loadSettingsFromJson(fileName)
fileName |
Name of the JSON file to load. |
Load a settings object from a JSON file, restoring object classes and attributes.
An R object as specified by the JSON.
Log a message at the DEBUG level
logDebug(...)
logDebug(...)
... |
Zero or more objects which can be coerced to character (and which are pasted together with no separator). |
Log a message at the specified level. The message will be sent to all the registered loggers.
Log a message at the ERROR level
logError(...)
logError(...)
... |
Zero or more objects which can be coerced to character (and which are pasted together with no separator). |
Log a message at the specified level. The message will be sent to all the registered loggers.
Log a message at the FATAL level
logFatal(...)
logFatal(...)
... |
Zero or more objects which can be coerced to character (and which are pasted together with no separator). |
Log a message at the specified level. The message will be sent to all the registered loggers. This
function is be automatically called when an error occurs, and should not be called directly. Use
stop()
instead.
Log a message at the INFO level
logInfo(...)
logInfo(...)
... |
Zero or more objects which can be coerced to character (and which are pasted together with no separator). |
Log a message at the specified level. The message will be sent to all the registered loggers. This
is equivalent to calling R's native message()
function.
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
Log a message at the TRACE level
logTrace(...)
logTrace(...)
... |
Zero or more objects which can be coerced to character (and which are pasted together with no separator). |
Log a message at the specified level. The message will be sent to all the registered loggers.
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
Log a message at the WARN level
logWarn(...)
logWarn(...)
... |
Zero or more objects which can be coerced to character (and which are pasted together with no separator). |
Log a message at the specified level. The message will be sent to all the registered loggers. This
function is automatically called when a warning is thrown, and should not be called directly. Use
warning()
instead.
Create a cluster of nodes for parallel computation
makeCluster( numberOfThreads, singleThreadToMain = TRUE, setAndromedaTempFolder = TRUE )
makeCluster( numberOfThreads, singleThreadToMain = TRUE, setAndromedaTempFolder = TRUE )
numberOfThreads |
Number of parallel threads. |
singleThreadToMain |
If |
setAndromedaTempFolder |
When TRUE, the andromedaTempFolder option will be copied to each thread. |
An object representing the cluster.
fun <- function(x) { return (x^2) } cluster <- makeCluster(numberOfThreads = 3) clusterApply(cluster, 1:10, fun) stopCluster(cluster)
fun <- function(x) { return (x^2) } cluster <- makeCluster(numberOfThreads = 3) clusterApply(cluster, 1:10, fun) stopCluster(cluster)
In a list of object of the same type, find those that match the input
matchInList(x, toMatch)
matchInList(x, toMatch)
x |
A list of objects of the same type. |
toMatch |
The object to match. |
Typically, toMatch will contain a subset of the variables that are in the objects in the list. Any
object matching all variables in toMatch
will be included in the result.
A list of objects that match the toMatch
object.
x <- list( a = list(name = "John", age = 25, gender = "M"), b = list(name = "Mary", age = 24, gender = "F") ) matchInList(x, list(name = "Mary")) # $a # $a$name # [1] "John" # # $a$age # [1] 25 # # # $b # $b$name # [1] "Mary" # # $b$age # [1] 24
x <- list( a = list(name = "John", age = 25, gender = "M"), b = list(name = "Mary", age = 24, gender = "F") ) matchInList(x, list(name = "Mary")) # $a # $a$name # [1] "John" # # $a$age # [1] 25 # # # $b # $b$name # [1] "Mary" # # $b$age # [1] 24
Register a logger
registerLogger(logger)
registerLogger(logger)
logger |
An object of type |
Registers a logger as created using the createLogger
function to the logging system.
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
Save a settings object as JSON file
saveSettingsToJson(object, fileName)
saveSettingsToJson(object, fileName)
object |
R object to be saved. |
fileName |
File name where the object should be saved. |
Save a setting object as a JSON file, using pretty formatting and preserving object classes and attributes.
Select variables from a list of objects of the same type
selectFromList(x, select)
selectFromList(x, select)
x |
A list of objects of the same type. |
select |
A character vector of names of variables to select. |
x <- list( a = list(name = "John", age = 25, gender = "M"), b = list(name = "Mary", age = 24, gender = "F") ) selectFromList(x, c("name", "age")) # $a # $a$name # [1] "John" # # $a$age # [1] 25 # # # $b # $b$name # [1] "Mary" # # $b$age # [1] 24
x <- list( a = list(name = "John", age = 25, gender = "M"), b = list(name = "Mary", age = 24, gender = "F") ) selectFromList(x, c("name", "age")) # $a # $a$name # [1] "John" # # $a$age # [1] 25 # # # $b # $b$name # [1] "Mary" # # $b$age # [1] 24
Stop the cluster
stopCluster(cluster)
stopCluster(cluster)
cluster |
The cluster to stop |
fun <- function(x) { return (x^2) } cluster <- makeCluster(numberOfThreads = 3) clusterApply(cluster, 1:10, fun) stopCluster(cluster)
fun <- function(x) { return (x^2) } cluster <- makeCluster(numberOfThreads = 3) clusterApply(cluster, 1:10, fun) stopCluster(cluster)
Unregister a logger
unregisterLogger(x, silent = FALSE)
unregisterLogger(x, silent = FALSE)
x |
Can either be an integer (e.g. 2 to remove the second logger), the name of the logger, or the logger object itself. |
silent |
If TRUE, no warning will be issued if the logger is not found. |
Unregisters a logger from the logging system.
Returns TRUE if the logger was removed.
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")
appender <- createConsoleAppender(layout = layoutTimestamp) logger <- createLogger(name = "SIMPLE", threshold = "INFO", appenders = list(appender)) registerLogger(logger) logTrace("This event is below the threshold (INFO)") logInfo("Hello world") unregisterLogger("SIMPLE")