rm(list = ls())
seed <- 1
set.seed(seed)
# Load libraries
require(kaiaulu)
require(data.table)
require(yaml)
require(stringi)
require(XML)
require(httr)
require(gt)Open source projects require a means for developers to communicate. These may include mailing lists, issue trackers, discord, etc. This notebooks showcases how to download data from mailing list archives.
Mailing lists are typically organized by topic or purpose. For example, the OpenSSL project maintains several mailing lists, each serving a different purpose:
The same mailing list, such as project-users may be stored in different archives. For example, there may be two archive mailing lists covering different time ranges. The data may overlap, or be incomplete depending on the archive type. Sometimes, each archive may come in a different format, communication may result from bots, etc. Different formats require different code to both download and parse them. Therefore, if the intent is to obtain a comprehensive understanding of the mailing list communication of a project, manual inspection of the available archives and their time range is strongly recommended.
Careful manual inspection of the mailing list data of particular importance when using Kaiaulu metrics such as social smells, where a incomplete mailing list archive will result in erroneous metric calculation (i.e. absence of communication reported in a metric when developing a feature being due to missing data, instead not ocuring within the data).
Two often used archive types are mod_mbox and pipermail, which Kaiaulu offer functions to download data from. The former is commonly used by the Apache Software Foundation projects, such as Apache Geronimo. The latter, is more commonly use in GNU related projects, such as OpenSSL, but this can vary.
Mod Mbox archives also organize mailing lists by topic. The apache mailing list archives can be found at https://lists.apache.org/.
This notebook demonstrates how to download and refresh mailing list archives from Mod Mbox and Pipermail.
Mailing List archives are hosted by their respective open source projects. Therefore, in order to use Kaiaulu downloaders to obtain mail data, you will need to access the respective open source project, and find out the URL tied to the archive you are interested in. Generally, that is the developer mailing list, if your interest is to understand communication patterns among developers. Alternatively, if the focus of the research is Q&A from the user base, then a user mailing list may make more sense.
Because project lifetime can go as far as a few decades, to have the full picture of what communication took place in the project you may need to download multiple archives and combine them, after turning them into tables using the Kaiaulu parser.
The information you need to find out for each open source project is documented in Kaiaulu using a project configuration file format. For pipermail and mod_mbox this is as follows:
mailing_list:
# for pipermail
pipermail:
project_key_1:
mailing_list: https://mta.openssl.org/pipermail/openssl-users/
save_folder_path: ../../rawdata/helix/pipermail/save_mbox_mail
# for mod mbox
mod_mbox:
project_key_1:
mailing_list: https://lists.apache.org/list.html?announce@apache.org
save_folder_path: ../../rawdata/helix/mod_mbox/save_mbox_mailThe most manual time intensive step you will be required is to locate the URL of the mailing list archive you wish for in the project website. Once located, this is specified in Kaiaulu’s project configuration file under mailing_list. Note for pipermail this URL should point to the page containing links to the monthly archives (e.g. https://mta.openssl.org/pipermail/openssl-users/), not the top-level mailing list page that contains all the different types of archives (e.g. https://mta.openssl.org/mailman/listinfo/).
Regardless of which mail archive you choose, the downloaders will store the mail data in monthly files, in a .mbox format. This is a simple text file that contains some markings to identify the header of the e-mail containing title, authors, etc. While you can open any of the .mbox downloaded files with any text editor, Kaiaulu parsers will format them into tables, as demonstrated below.
In addition to the mailing list configurations, you need to specify the path to the Perceval binary in your computer in Kaiaulu’s tools.yml. This is so Kaiaulu can make a system call to the tool to parse the data. See the wiki for further details on how to setup third party tools.
# Load tools configuration
tools <- parse_config("../tools.yml")
parse_perceval_path <- get_tool_project("perceval", tools)
# Load project configuration
conf <- parse_config("../conf/helix.yml")
mbox_file_path <- get_mbox_path(conf, "project_key_1")With the configurations loaded, we can proceed to download the mailing list archives. The downloaders are responsible for fetching the archives from the specified mailing lists and saving them locally in .mbox format.
For Pipermail, we need to specify the project key, which is used to retrieve the configuration parameters for the specific project. The project key is used to identify the project in the configuration file.
Now, we can use the getter functions to retrieve the configuration parameters for the specified project key.
conf <- parse_config("../conf/openssl.yml")
pipermail_mailing_list <- get_pipermail_domain(conf, "project_key_1")
pipermail_save_folder_path <- get_pipermail_path(conf, "project_key_1")
# Define the date range
pipermail_start_year_month <- 202310
pipermail_end_year_month <- 202405With our configurations loaded, we can proceed to downloading the mailing list archives.
# Download archives
download_pipermail(
mailing_list = pipermail_mailing_list,
start_year_month = pipermail_start_year_month,
end_year_month = pipermail_end_year_month,
save_folder_path = pipermail_save_folder_path,
verbose = TRUE
)After running this function, the .mbox files will be saved in the specified directory with filenames like 202310.mbox, 202311.mbox, etc, which can be parsed in a table:
parsed_mail <- parse_mbox(
perceval_path = parse_perceval_path,
mbox_file_path = pipermail_save_folder_path
)
parsed_mail %>%
head(10) %>%
gt()| reply_id | in_reply_to_id | reply_datetimetz | reply_from | reply_subject | reply_body |
|---|---|---|---|---|---|
| <CAHnN3bfgrfE0frt_E4DK2OiQ7rfNQ0fY2w1avcxW0XP-kNUFKA@mail.gmail.com> | NA | Mon, 2 Oct 2023 09:41:57 +0200 | thomas.bailleux at sandboxquantum.com (Thomas Bailleux) | OpenSSL 3, "reuse" capability in `d2i_TYPE_BIO` and documentation | Hello OpenSSL, I'm currently migrating a codebase from OpenSSL 1.1.1 to OpenSSL 3. Since I may use OpenSSL providers in the future, I decided to use these new `_ex` functions from OpenSSL 3. While reading the "Old functions that should be changed" from the migration guide[1], I came across an oddity: it is claimed that in order to use a non-default library context when parsing an `X509` or an `EVP_PKEY`, `TYPE_new_ex` must be used (e.g. `X509_new_ex`), and then we have to use the "reuse" capability from the various parsing functions (`PEM_read_bio_X509`): Some functions can be passed an object that has already been set up with a > library context such as d2i_X509(3), d2i_X509_CRL(3), d2i_X509_REQ(3) and > d2i_X509_PUBKEY(3). If NULL is passed instead then the created object will > be set up with the default library context. Use X509_new_ex(3), > X509_CRL_new_ex(3), X509_REQ_new_ex(3) and X509_PUBKEY_new_ex(3) if a > library context is required. So basically we have to do the following: BIO *bio; > OSSL_LIB_CTX* lib_ctx; > X509 *x509 = X509_new_ex(lib_ctx, NULL); > if (d2i_X509_bio(bio, &x509) != NULL) { > // success > } else { > // error > } > However, in the `D2I_X509` manpage[2], the following is stated: On a successful return, if **a* is not NULL then it is assumed that **a* > contains a valid *TYPE* structure and an attempt is made to reuse it. > This "reuse" capability is present for historical compatibility but its use > is *strongly discouraged* (see BUGS below, and the discussion in the > RETURN VALUES section). > ? > BUGS > ? > > As a result of the above issues the "reuse" behaviour is strongly > discouraged. > So if I'm understanding correctly, this "reuse" capability is discouraged, still present for historical compatibility, but in OpenSSL 3 we have to use it if we want to use a custom library context. This divergence between these two bits of documentation bothers me. Do you have an opinion on this? Regards, - thomas [1]: https://www.openssl.org/docs/man3.1/man7/migration_guide.html#Using-a-Library-Context---Old-functions-that-should-be-changed [2]: https://www.openssl.org/docs/man3.1/man3/d2i_X509.html -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20231002/858adccf/attachment.htm> |
| <CAFRq_FA0EE046w53PEORTb78iC8C1sb2OaCHAuRMECj2EAzKwA@mail.gmail.com> | NA | Mon, 2 Oct 2023 16:01:42 +0300 | liad.dekel at broadcom.com (Liad Dekel) | Using OpenSSL with Devcrypto in async mode | Hi All, This is my first time sending mail to a mailing list, thus if the message is not in the correct format my apology. I'm using OpenSSL together with Devcrypto in order to utilize crypto hardware acceleration. I just found that Devcrypto has an asynch mode in which it's queueing the crypto requests received from userspace. I wonder whether OpenSSL can work with Devcrypto in async mode. From a short research of the source code, seems like it's not - Devcrypto requires a "special" ioctl commands for async operations (CIOCASYNCCRYPT, CIOCASYNCFETCH)which are not exist in e_devcrypto.c Many thanks, Liad -- This electronic communication and the information and any files transmitted with it, or attached to it, are confidential and are intended solely for the use of the individual or entity to whom it is addressed and may contain information that is confidential, legally privileged, protected by privacy laws, or otherwise restricted from disclosure to anyone else. If you are not the intended recipient or the person responsible for delivering the e-mail to the intended recipient, you are hereby notified that any use, copying, distributing, dissemination, forwarding, printing, or copying of this e-mail is strictly prohibited. If you received this e-mail in error, please return the e-mail to the sender, delete it from your computer, and destroy any printed copy of it. -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20231002/096e8f0b/attachment.htm> |
| <3e8f1ab7955b365946c9f50e21084c2bd1f8f655.camel@openssl.org> | <CAHnN3bfgrfE0frt_E4DK2OiQ7rfNQ0fY2w1avcxW0XP-kNUFKA@mail.gmail.com> | Tue, 03 Oct 2023 15:47:41 +0200 | tomas at openssl.org (Tomas Mraz) | OpenSSL 3, "reuse" capability in `d2i_TYPE_BIO` and documentation | Hello Thomas, I've created a pull request that should clarify the matter: https://github.com/openssl/openssl/pull/22265 Please look there. Tomas Mraz, OpenSSL On Mon, 2023-10-02 at 09:41 +0200, Thomas Bailleux wrote: > Hello OpenSSL, > > I'm currently migrating a codebase from OpenSSL 1.1.1 to OpenSSL 3. > Since I may use OpenSSL providers in the future, I decided to use > these new `_ex` functions from OpenSSL 3. > > While reading the "Old functions that should be changed" from the > migration guide[1], I came across an oddity: it is claimed that in > order to use a non-default library context when parsing an `X509` or > an `EVP_PKEY`, `TYPE_new_ex` must be used (e.g. `X509_new_ex`), and > then we have to use the "reuse" capability from the various parsing > functions (`PEM_read_bio_X509`): > > > Some functions can be passed an object that has already been set up > > with a library context such as d2i_X509(3), d2i_X509_CRL(3), > > d2i_X509_REQ(3) and d2i_X509_PUBKEY(3). If NULL is passed instead > > then the created object will be set up with the default library > > context. Use X509_new_ex(3), X509_CRL_new_ex(3), X509_REQ_new_ex(3) > > and X509_PUBKEY_new_ex(3) if a library context is required. > > So basically we have to do the following: > > > BIO *bio; > > OSSL_LIB_CTX* lib_ctx; > > X509 *x509 = X509_new_ex(lib_ctx, NULL); > > if (d2i_X509_bio(bio, &x509) != NULL) { > > ? // success > > } else { > > ? // error > > } > > > > > However, in the `D2I_X509` manpage[2], the following is stated: > > > On a successful return, if *a is not NULL then it is assumed that > > *a contains a valid TYPE structure and an attempt is made to reuse > > it. This "reuse" capability is present for historical compatibility > > but its use is strongly discouraged (see BUGS below, and the > > discussion in the RETURN VALUES section). > > ? > > BUGS > > ? > > As a result of the above issues the "reuse" behaviour is strongly > > discouraged. > > > > So if I'm understanding correctly, this "reuse" capability is > discouraged, still present for historical compatibility, but in > OpenSSL 3 we have to use it if we want to use a custom library > context. > > This divergence between these two bits of documentation bothers me. > Do you have an opinion on this? > > Regards, > > - thomas > > > [1]: > https://www.openssl.org/docs/man3.1/man7/migration_guide.html#Using-a-Library-Context---Old-functions-that-should-be-changed > [2]: https://www.openssl.org/docs/man3.1/man3/d2i_X509.html -- Tom?? Mr?z, OpenSSL |
| <CAHnN3bddNxRWKktVKfO6VT29k6QDBHP36iEWg23GMK5ppMGykQ@mail.gmail.com> | <3e8f1ab7955b365946c9f50e21084c2bd1f8f655.camel@openssl.org> | Wed, 4 Oct 2023 09:36:50 +0200 | thomas.bailleux at sandboxquantum.com (Thomas Bailleux) | OpenSSL 3, "reuse" capability in `d2i_TYPE_BIO` and documentation | Hi Tomas, Thanks for your reply. I wonder if we shouldn't be providing a new API for X509 instead. We already have `PEM_read_bio_PrivateKey_ex` and `d2i_PrivateKey_ex_bio`, what about having `PEM_read_bio_X509_ex` and `d2i_X509_ex_bio` too? I feel like the documentation can be confusing with all these conditions (OK if using OSSL_LIB_CTX, otherwise not OK, and in both cases *x is freed even on error etc). Also, we can move this discussion to the PR you just opened if you prefer. Regards, - thomas On Tue, Oct 3, 2023 at 3:47?PM Tomas Mraz <tomas at openssl.org> wrote: > Hello Thomas, > > I've created a pull request that should clarify the matter: > > https://github.com/openssl/openssl/pull/22265 > > Please look there. > > Tomas Mraz, OpenSSL > > On Mon, 2023-10-02 at 09:41 +0200, Thomas Bailleux wrote: > > Hello OpenSSL, > > > > I'm currently migrating a codebase from OpenSSL 1.1.1 to OpenSSL 3. > > Since I may use OpenSSL providers in the future, I decided to use > > these new `_ex` functions from OpenSSL 3. > > > > While reading the "Old functions that should be changed" from the > > migration guide[1], I came across an oddity: it is claimed that in > > order to use a non-default library context when parsing an `X509` or > > an `EVP_PKEY`, `TYPE_new_ex` must be used (e.g. `X509_new_ex`), and > > then we have to use the "reuse" capability from the various parsing > > functions (`PEM_read_bio_X509`): > > > > > Some functions can be passed an object that has already been set up > > > with a library context such as d2i_X509(3), d2i_X509_CRL(3), > > > d2i_X509_REQ(3) and d2i_X509_PUBKEY(3). If NULL is passed instead > > > then the created object will be set up with the default library > > > context. Use X509_new_ex(3), X509_CRL_new_ex(3), X509_REQ_new_ex(3) > > > and X509_PUBKEY_new_ex(3) if a library context is required. > > > > So basically we have to do the following: > > > > > BIO *bio; > > > OSSL_LIB_CTX* lib_ctx; > > > X509 *x509 = X509_new_ex(lib_ctx, NULL); > > > if (d2i_X509_bio(bio, &x509) != NULL) { > > > // success > > > } else { > > > // error > > > } > > > > > > > > > However, in the `D2I_X509` manpage[2], the following is stated: > > > > > On a successful return, if *a is not NULL then it is assumed that > > > *a contains a valid TYPE structure and an attempt is made to reuse > > > it. This "reuse" capability is present for historical compatibility > > > but its use is strongly discouraged (see BUGS below, and the > > > discussion in the RETURN VALUES section). > > > ? > > > BUGS > > > ? > > > As a result of the above issues the "reuse" behaviour is strongly > > > discouraged. > > > > > > > So if I'm understanding correctly, this "reuse" capability is > > discouraged, still present for historical compatibility, but in > > OpenSSL 3 we have to use it if we want to use a custom library > > context. > > > > This divergence between these two bits of documentation bothers me. > > Do you have an opinion on this? > > > > Regards, > > > > - thomas > > > > > > [1]: > > > https://www.openssl.org/docs/man3.1/man7/migration_guide.html#Using-a-Library-Context---Old-functions-that-should-be-changed > > [2]: https://www.openssl.org/docs/man3.1/man3/d2i_X509.html > > -- > Tom?? Mr?z, OpenSSL > > -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20231004/02a861c4/attachment.htm> |
| <d15be4ca9cc5c4ebfcf41b4f56cac04aa0c33b68.camel@openssl.org> | <CAHnN3bddNxRWKktVKfO6VT29k6QDBHP36iEWg23GMK5ppMGykQ@mail.gmail.com> | Wed, 04 Oct 2023 09:39:34 +0200 | tomas at openssl.org (Tomas Mraz) | OpenSSL 3, "reuse" capability in `d2i_TYPE_BIO` and documentation | Clearly we need the documentation fix. If there should be a new API or not that's a separate discussion. If you want to propose it, please open a new GH feature request issue. Regards, Tomas Mraz, OpenSSL On Wed, 2023-10-04 at 09:36 +0200, Thomas Bailleux wrote: > Hi Tomas, > > Thanks for your reply. > > I wonder if we shouldn't be providing a new API for X509 instead. We > already have `PEM_read_bio_PrivateKey_ex` and > `d2i_PrivateKey_ex_bio`, what about having `PEM_read_bio_X509_ex` and > `d2i_X509_ex_bio` too? > I feel like the documentation can be confusing with all these > conditions (OK if using OSSL_LIB_CTX, otherwise not OK, and in both > cases *x is freed even on error etc). > > Also, we can move this discussion to the PR you just opened if you > prefer. > > Regards, > > - thomas > > On Tue, Oct 3, 2023 at 3:47?PM Tomas Mraz <tomas at openssl.org> wrote: > > Hello Thomas, > > > > I've created a pull request that should clarify the matter: > > > > https://github.com/openssl/openssl/pull/22265 > > > > Please look there. > > > > Tomas Mraz, OpenSSL > > > > On Mon, 2023-10-02 at 09:41 +0200, Thomas Bailleux wrote: > > > Hello OpenSSL, > > > > > > I'm currently migrating a codebase from OpenSSL 1.1.1 to OpenSSL > > > 3. > > > Since I may use OpenSSL providers in the future, I decided to use > > > these new `_ex` functions from OpenSSL 3. > > > > > > While reading the "Old functions that should be changed" from the > > > migration guide[1], I came across an oddity: it is claimed that > > > in > > > order to use a non-default library context when parsing an `X509` > > > or > > > an `EVP_PKEY`, `TYPE_new_ex` must be used (e.g. `X509_new_ex`), > > > and > > > then we have to use the "reuse" capability from the various > > > parsing > > > functions (`PEM_read_bio_X509`): > > > > > > > Some functions can be passed an object that has already been > > > > set up > > > > with a library context such as d2i_X509(3), d2i_X509_CRL(3), > > > > d2i_X509_REQ(3) and d2i_X509_PUBKEY(3). If NULL is passed > > > > instead > > > > then the created object will be set up with the default library > > > > context. Use X509_new_ex(3), X509_CRL_new_ex(3), > > > > X509_REQ_new_ex(3) > > > > and X509_PUBKEY_new_ex(3) if a library context is required. > > > > > > So basically we have to do the following: > > > > > > > BIO *bio; > > > > OSSL_LIB_CTX* lib_ctx; > > > > X509 *x509 = X509_new_ex(lib_ctx, NULL); > > > > if (d2i_X509_bio(bio, &x509) != NULL) { > > > > ? // success > > > > } else { > > > > ? // error > > > > } > > > > > > > > > > > > > However, in the `D2I_X509` manpage[2], the following is stated: > > > > > > > On a successful return, if *a is not NULL then it is assumed > > > > that > > > > *a contains a valid TYPE structure and an attempt is made to > > > > reuse > > > > it. This "reuse" capability is present for historical > > > > compatibility > > > > but its use is strongly discouraged (see BUGS below, and the > > > > discussion in the RETURN VALUES section). > > > > ? > > > > BUGS > > > > ? > > > > As a result of the above issues the "reuse" behaviour is > > > > strongly > > > > discouraged. > > > > > > > > > > So if I'm understanding correctly, this "reuse" capability is > > > discouraged, still present for historical compatibility, but in > > > OpenSSL 3 we have to use it if we want to use a custom library > > > context. > > > > > > This divergence between these two bits of documentation bothers > > > me. > > > Do you have an opinion on this? > > > > > > Regards, > > > > > > - thomas > > > > > > > > > [1]: > > > https://www.openssl.org/docs/man3.1/man7/migration_guide.html#Using-a-Library-Context---Old-functions-that-should-be-changed > > > [2]: https://www.openssl.org/docs/man3.1/man3/d2i_X509.html > > -- Tom?? Mr?z, OpenSSL |
| <CAHnN3bcEUMfCoaYE0pTvRKh2TRSPV75_vbnaiGpXhx4eZqQg1w@mail.gmail.com> | NA | Thu, 5 Oct 2023 09:39:22 +0200 | thomas.bailleux at sandboxaq.com (Thomas Bailleux) | `core_obj_create` and multi-threaded applications | Hello OpenSSL, I'm currently writing a multi-threaded application using OpenSSL 3.1. For some reason, I've decided to instantiate a `OSSL_LIB_CTX` per thread. I'm also trying the provider API with oqs-provider, and I ran into an issue that is caused by the fact that `core_obj_create` is called by `OSSL_PROVIDER_load` which itself is called by multiple threads running at the same time. I described the bug in the following thread: https://github.com/open-quantum-safe/oqs-provider/issues/272 Basically, the oqs-provider init function registers some custom OIDs using the `core_obj_create` which makes a call to `OBJ_create`. As described in the BUGS section of the `OBJ_create` man page[1] of *OpenSSL 3.0*, `OBJ_create` used to be non thread-safe. However, in *OpenSSL 3.1*[2], it is thread safe: These functions were not thread safe in OpenSSL 3.0 and before. > So I think there is a race condition in `core_obj_create`[3] function: the function checks if the OID already exists, if so it returns OK (1), otherwise it makes a call to `OBJ_create`. The TOCTOU is happening between these two statements basically. What do you suggest to mitigate this issue? Regards, - thomas [1]: https://www.openssl.org/docs/man3.0/man3/OBJ_create.html [2]: https://www.openssl.org/docs/man3.1/man3/OBJ_create.html [3]: https://github.com/openssl/openssl/blob/9c20f5db0feaddc4c9ea4c4b2b07e6d87d6701f1/crypto/provider_core.c#L2127-L2133 -- Thomas Bailleux Cryptography software engineer thomas at sandboxaq.com +33 6 47 04 61 58 | Paris, FRANCE -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20231005/65bbbdfa/attachment.htm> |
| <521386eb-6b6b-498a-8962-97d68b81eab2@openssl.org> | <CAHnN3bcEUMfCoaYE0pTvRKh2TRSPV75_vbnaiGpXhx4eZqQg1w@mail.gmail.com> | Thu, 5 Oct 2023 10:51:44 +0100 | matt at openssl.org (Matt Caswell) | `core_obj_create` and multi-threaded applications | It mostly seems to cause an inconsistency in that sometimes core_obj_create returns 1 if the OID already exists and sometimes it returns 0. Really core_obj_create should be a thin wrapper around OBJ_create. It probably shouldn't do the initial check at all and just call OBJ_create - which would make it always consistent, i.e. return 0 if it already exists. oqsprovider probably needs to be tolerant of this failure condition, i.e. it doesn't care if it already exists - so ignore a 0 return. Matt On 05/10/2023 08:39, Thomas Bailleux wrote: > Hello OpenSSL, > > I'm currently writing a multi-threaded application using OpenSSL 3.1. > For some reason, I've decided to instantiate a `OSSL_LIB_CTX` per thread. > I'm also trying the provider API with oqs-provider, and I ran into an > issue that is caused > by the fact that `core_obj_create` is called by `OSSL_PROVIDER_load` > which itself is called by multiple threads running at the same time. > > I described the bug in the following thread: > https://github.com/open-quantum-safe/oqs-provider/issues/272 > <https://github.com/open-quantum-safe/oqs-provider/issues/272> > > Basically, the oqs-provider init function registers some custom OIDs > using the `core_obj_create` which makes a call to `OBJ_create`. > > As described in the BUGS section of the `OBJ_create` man page[1] of > *OpenSSL 3.0*, `OBJ_create` used to be non thread-safe. > However, in *OpenSSL 3.1*[2], it is thread safe: > > These functions were not thread safe in OpenSSL 3.0 and before. > > > So I think there is a race condition in `core_obj_create`[3] function: > the function checks if the OID already exists, if so it returns OK (1), > otherwise it makes a call to `OBJ_create`. The TOCTOU is happening > between these two statements basically. > > What do you suggest to mitigate this issue? > > Regards, > > - thomas > > [1]: https://www.openssl.org/docs/man3.0/man3/OBJ_create.html > <https://www.openssl.org/docs/man3.0/man3/OBJ_create.html> > [2]: https://www.openssl.org/docs/man3.1/man3/OBJ_create.html > <https://www.openssl.org/docs/man3.1/man3/OBJ_create.html> > [3]: > https://github.com/openssl/openssl/blob/9c20f5db0feaddc4c9ea4c4b2b07e6d87d6701f1/crypto/provider_core.c#L2127-L2133 <https://github.com/openssl/openssl/blob/9c20f5db0feaddc4c9ea4c4b2b07e6d87d6701f1/crypto/provider_core.c#L2127-L2133> > > -- > > Thomas Bailleux > > Cryptography software engineer > > thomas at sandboxaq.com <mailto:thomas at sandboxaq.com> > > +33 6 47 04 61 58 | Paris, FRANCE > > |
| <11e418ac-c325-83b2-f283-10d872b9db08@tana.it> | NA | Fri, 6 Oct 2023 11:04:31 +0200 | vesely at tana.it (Alessandro Vesely) | How to use EVP_DigestSignInit[_ex]()? | Hi all, my scarce crypto knowledge notwithstanding, I'm trying to maintain a DKIM signing function. With the previous openssl version, I added a snippet to sign using ed25519, more or less like so: char *digest; // digest already computed from size_t diglen; // normalized message header EVP_PKEY *crypto_pkey; // given size_t crypto_outlen; // expected unsigned char *crypto_out; EVP_MD_CTX *md_ctx = EVP_MD_CTX_new(); if (md_ctx == NULL) goto error; int status = EVP_DigestSignInit(md_ctx, NULL, NULL, NULL, crypto_pkey); if (status == 1) status = EVP_DigestSign(md_ctx, crypto_out, &crypto_outlen, digest, diglen); EVP_MD_CTX_free(md_ctx); if (status != 1) goto error; That works. With the onset of openssl 3 and RSA_sign() deprecation, I thought that snippet could work with RSA as well. Given an RSA key, that code returns no error but a NULL crypto_out. I should have specified SHA256 or (deprecated) SHA1, but I don't know how to obtain an EVP_MD *type. Or is it better to use the _ex version? Best Ale -- |
| <4c071c8f-aee6-1f3a-05ec-5044195e8c19@tana.it> | <11e418ac-c325-83b2-f283-10d872b9db08@tana.it> | Fri, 6 Oct 2023 19:54:10 +0200 | vesely at tana.it (Alessandro Vesely) | How to use EVP_DigestSignInit[_ex]()? | I fixed my code as follows: On Fri 06/Oct/2023 11:04:31 +0200 Alessandro Vesely wrote: > > my scarce crypto knowledge notwithstanding, I'm trying to maintain a DKIM > signing function.? With the previous openssl version, I added a snippet to sign > using ed25519, more or less like so: > > ??? char *digest;????????????? // digest already computed from > ??? size_t diglen;???????????? // normalized message header > > ??? EVP_PKEY *crypto_pkey;???? // given > > ??? size_t crypto_outlen;????? // expected > ??? unsigned char *crypto_out; > > ??? EVP_MD_CTX *md_ctx = EVP_MD_CTX_new(); > ??? if (md_ctx == NULL) goto error; int status; if (signalg != ED25519SHA256) status = EVP_DigestSignInit_ex(md_ctx, NULL, signalg == SHA1? "SHA1": "sha256", NULL, NULL, crypto_pkey, NULL); else status = EVP_DigestSignInit(md_ctx, NULL, NULL, NULL, crypto->crypto_pkey); > ??? if (status == 1) > ??????? status = EVP_DigestSign(md_ctx, > ??????????? crypto_out, &crypto_outlen, > ??????????? digest, diglen); > ??? EVP_MD_CTX_free(md_ctx); > ??? if (status != 1) goto error; Now, that works well with ED25519. With RSA keys, all tests fail, delivering a signature different than expected. Can that be because I still don't have set RSA_PKCS1_PADDING? How could that be done? Best Ale |
| <CAO5S7GhapfihSC6SpN9-mX9xE31feG_UJ0e3QV9m9NPxNpXkbA@mail.gmail.com> | NA | Mon, 9 Oct 2023 12:21:25 +0530 | brahmaji.k at gmail.com (Brahmaji K) | Request for Openssl APIs to be used to sort the certificate chain | Hi Team, Could you please help provide Openssl APIs (or list of APIs) to be used to sort the certificates used in the certificate chain? -------------- next part -------------- An HTML attachment was scrubbed... URL: <https://mta.openssl.org/pipermail/openssl-users/attachments/20231009/c1c38df4/attachment.htm> |
The download_mod_mbox() function downloads Mod Mbox archives from a specified Apache Pony Mail mailing list over a given date range. We obtain the required parameters from the project configuration file, as done before:
conf <- parse_config("../conf/helix.yml")
mbox_mailing_list <- get_mbox_domain(conf, "project_key_1")
mbox_save_folder_path <- get_mbox_path(conf, "project_key_1")
# Define the date range
mbox_start_year_month <- 202310
mbox_end_year_month <- 202405The start_year_month and end_year_month time range parameters should be set manually, as with Pipermail.
download_mod_mbox(
mailing_list = mbox_mailing_list,
start_year_month = mbox_start_year_month,
end_year_month = mbox_end_year_month,
save_folder_path = mbox_save_folder_path,
verbose = TRUE
)After running the function, it constructs URLs like: https://lists.apache.org/api/mbox.lua?list=announce@apache.org&date=2024-01 and saves the files in the specified folder.
parsed_mail <- parse_mbox(
perceval_path = parse_perceval_path,
mbox_file_path = mbox_file_path
)
parsed_mail %>%
head(10) %>%
gt()| reply_id | in_reply_to_id | reply_datetimetz | reply_from | reply_to | reply_cc | reply_subject | reply_body |
|---|---|---|---|---|---|---|---|
| <CAPoyBqQqpVDeCHZ8DkmLUTtthNZN269Eqe8HJmvjpPcJ3vSiwA@mail.gmail.com> | NA | Tue, 16 Oct 2012 18:51:17 +0200 | Olivier Lamy <olamy@apache.org> | dev@helix.incubator.apache.org | NA | Test | simple test |
| <CABaj-QYsXFCAvJO+PV2d-azasi5Vq72QeyMud3gnsmHEj0Qpmw@mail.gmail.com> | <CAPoyBqQqpVDeCHZ8DkmLUTtthNZN269Eqe8HJmvjpPcJ3vSiwA@mail.gmail.com> | Tue, 16 Oct 2012 09:53:11 -0700 | kishore g <g.kishore@gmail.com> | dev@helix.apache.org | dev@helix.incubator.apache.org | Re: Test | works for me On Tue, Oct 16, 2012 at 9:51 AM, Olivier Lamy <olamy@apache.org> wrote: > simple test > |
| <CALF7aMSpmfbmnE8-R_q8fohr=9CpbekfFXPyzk95zZhRw1K8-w@mail.gmail.com> | <CAPoyBqQqpVDeCHZ8DkmLUTtthNZN269Eqe8HJmvjpPcJ3vSiwA@mail.gmail.com> | Tue, 16 Oct 2012 09:54:00 -0700 | Mahadev Konar <mahadev@apache.org> | dev@helix.apache.org | dev@helix.incubator.apache.org | Re: Test | Working! On Tue, Oct 16, 2012 at 9:51 AM, Olivier Lamy <olamy@apache.org> wrote: > simple test |
| <CAPoyBqRVBH-h4BXshOTcXHiMfb-uSQQ0MtG6RgHLx6R6kx_ZMw@mail.gmail.com> | NA | Tue, 16 Oct 2012 19:05:42 +0200 | Olivier Lamy <olamy@apache.org> | dev <dev@helix.incubator.apache.org> | NA | Incubator status page started | Hi, As we have now a mailing list :-) FYI I have started the page: http://incubator.apache.org/projects/helix.html -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy |
| <CAPoyBqT3F7sxOEuftPhkKjqi7brf8XPD1d4McVWzxQXdzAcfvw@mail.gmail.com> | <CAPoyBqRVBH-h4BXshOTcXHiMfb-uSQQ0MtG6RgHLx6R6kx_ZMw@mail.gmail.com> | Tue, 16 Oct 2012 19:12:54 +0200 | Olivier Lamy <olamy@apache.org> | dev <dev@helix.incubator.apache.org> | NA | Re: Incubator status page started | So as I don't know if some folks from the proposal have or not an asf id. Can someone complete the committers list ? 2012/10/16 Olivier Lamy <olamy@apache.org>: > Hi, > As we have now a mailing list :-) > FYI I have started the page: http://incubator.apache.org/projects/helix.html > > > -- > Olivier Lamy > Talend: http://coders.talend.com > http://twitter.com/olamy | http://linkedin.com/in/olamy -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy |
| <CABaj-QZ8TSVJuFwPpZfDL6UH0GRh4xN+e9RHuv+EPBXOpp3t6g@mail.gmail.com> | <CAPoyBqT3F7sxOEuftPhkKjqi7brf8XPD1d4McVWzxQXdzAcfvw@mail.gmail.com> | Tue, 16 Oct 2012 10:22:54 -0700 | kishore g <g.kishore@gmail.com> | dev@helix.apache.org | dev <dev@helix.incubator.apache.org> | Re: Incubator status page started | Hi Olivier, Most of the folks are new to Apache. I will get their asf id list by end of today On Tue, Oct 16, 2012 at 10:12 AM, Olivier Lamy <olamy@apache.org> wrote: > So as I don't know if some folks from the proposal have or not an asf id. > Can someone complete the committers list ? > > > 2012/10/16 Olivier Lamy <olamy@apache.org>: > > Hi, > > As we have now a mailing list :-) > > FYI I have started the page: > http://incubator.apache.org/projects/helix.html > > > > > > -- > > Olivier Lamy > > Talend: http://coders.talend.com > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > > > -- > Olivier Lamy > Talend: http://coders.talend.com > http://twitter.com/olamy | http://linkedin.com/in/olamy > |
| <CAPoyBqS6YOOUzaNHVk9QvxvK_ce9cs+5=6bV7YJt5_7VGeH_1Q@mail.gmail.com> | <CABaj-QZ8TSVJuFwPpZfDL6UH0GRh4xN+e9RHuv+EPBXOpp3t6g@mail.gmail.com> | Tue, 16 Oct 2012 19:30:04 +0200 | Olivier Lamy <olamy@apache.org> | dev@helix.apache.org | NA | Re: Incubator status page started | Great ! Don't miss they need to send a cla too. -- Olivier Le 16 oct. 2012 19:23, "kishore g" <g.kishore@gmail.com> a écrit : > Hi Olivier, > > Most of the folks are new to Apache. I will get their asf id list by end of > today > > On Tue, Oct 16, 2012 at 10:12 AM, Olivier Lamy <olamy@apache.org> wrote: > > > So as I don't know if some folks from the proposal have or not an asf id. > > Can someone complete the committers list ? > > > > > > 2012/10/16 Olivier Lamy <olamy@apache.org>: > > > Hi, > > > As we have now a mailing list :-) > > > FYI I have started the page: > > http://incubator.apache.org/projects/helix.html > > > > > > > > > -- > > > Olivier Lamy > > > Talend: http://coders.talend.com > > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > > > > > > > -- > > Olivier Lamy > > Talend: http://coders.talend.com > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > > |
| <CABaj-QZCkD2RwJOYhG2hfkSsCQkF9kY3p-Fqjf2_+2WAR94+1Q@mail.gmail.com> | <CAPoyBqS6YOOUzaNHVk9QvxvK_ce9cs+5=6bV7YJt5_7VGeH_1Q@mail.gmail.com> | Tue, 16 Oct 2012 10:38:24 -0700 | kishore g <g.kishore@gmail.com> | dev <dev@helix.incubator.apache.org> | NA | Re: Incubator status page started | I think the reply to header is set incorrectly in the emails. it sends to dev@helix.apache.org On Tue, Oct 16, 2012 at 10:30 AM, Olivier Lamy <olamy@apache.org> wrote: > Great ! > Don't miss they need to send a cla too. > > -- > Olivier > Le 16 oct. 2012 19:23, "kishore g" <g.kishore@gmail.com> a écrit : > > > Hi Olivier, > > > > Most of the folks are new to Apache. I will get their asf id list by end > of > > today > > > > On Tue, Oct 16, 2012 at 10:12 AM, Olivier Lamy <olamy@apache.org> wrote: > > > > > So as I don't know if some folks from the proposal have or not an asf > id. > > > Can someone complete the committers list ? > > > > > > > > > 2012/10/16 Olivier Lamy <olamy@apache.org>: > > > > Hi, > > > > As we have now a mailing list :-) > > > > FYI I have started the page: > > > http://incubator.apache.org/projects/helix.html > > > > > > > > > > > > -- > > > > Olivier Lamy > > > > Talend: http://coders.talend.com > > > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > > > > > > > > > > > -- > > > Olivier Lamy > > > Talend: http://coders.talend.com > > > http://twitter.com/olamy | http://linkedin.com/in/olamy > > > > > > |
| <CAPoyBqTVk-QZF933ZDdWdP+xZP6GCH7rK27MrftSWRETjD6cFg@mail.gmail.com> | <CABaj-QZCkD2RwJOYhG2hfkSsCQkF9kY3p-Fqjf2_+2WAR94+1Q@mail.gmail.com> | Tue, 16 Oct 2012 21:24:36 +0200 | Olivier Lamy <olamy@apache.org> | dev@helix.incubator.apache.org | NA | Re: Incubator status page started | Good catch. I will check with infra. 2012/10/16 kishore g <g.kishore@gmail.com>: > I think the reply to header is set incorrectly in the emails. it sends to > dev@helix.apache.org > > On Tue, Oct 16, 2012 at 10:30 AM, Olivier Lamy <olamy@apache.org> wrote: > >> Great ! >> Don't miss they need to send a cla too. >> >> -- >> Olivier >> Le 16 oct. 2012 19:23, "kishore g" <g.kishore@gmail.com> a écrit : >> >> > Hi Olivier, >> > >> > Most of the folks are new to Apache. I will get their asf id list by end >> of >> > today >> > >> > On Tue, Oct 16, 2012 at 10:12 AM, Olivier Lamy <olamy@apache.org> wrote: >> > >> > > So as I don't know if some folks from the proposal have or not an asf >> id. >> > > Can someone complete the committers list ? >> > > >> > > >> > > 2012/10/16 Olivier Lamy <olamy@apache.org>: >> > > > Hi, >> > > > As we have now a mailing list :-) >> > > > FYI I have started the page: >> > > http://incubator.apache.org/projects/helix.html >> > > > >> > > > >> > > > -- >> > > > Olivier Lamy >> > > > Talend: http://coders.talend.com >> > > > http://twitter.com/olamy | http://linkedin.com/in/olamy >> > > >> > > >> > > >> > > -- >> > > Olivier Lamy >> > > Talend: http://coders.talend.com >> > > http://twitter.com/olamy | http://linkedin.com/in/olamy >> > > >> > >> -- Olivier Lamy Talend: http://coders.talend.com http://twitter.com/olamy | http://linkedin.com/in/olamy |
| <CABaj-QYBmpgVT5PMpoBZT4tL=6LwuLj4uz3FkFJQV9nCi0=75w@mail.gmail.com> | <CAPoyBqRNaqL7d=_7Wkbxw+W6E3qxB0myxzixctpsDtrXYiv9zw@mail.gmail.com> | Tue, 16 Oct 2012 12:29:57 -0700 | kishore g <g.kishore@gmail.com> | dev@helix.incubator.apache.org | NA | Re: test | What is the best way for others to subscribe. will dev-subscribe@helix.incubator.apache.org work? On Tue, Oct 16, 2012 at 12:26 PM, Olivier Lamy <olamy@apache.org> wrote: > sorry for noise! > |
Kaiaulu offers convenient function to add new e-mails since the last execution of the downloaders. These are defined as “refresh_*” functions. The most recent file name’s timestamp, which captures the latest month, is used as a starting date to download new files. The most recent file is deleted and re-downloaded to ensure all e-mails of the last month were downloaded, and subsequent files are then downloaded.
# Refresh archives
refresh_pipermail(
mailing_list = pipermail_mailing_list,
start_year_month = pipermail_start_year_month,
save_folder_path = pipermail_save_folder_path,
verbose = TRUE
)A similar function is also available for mod_mbox:
refresh_mod_mbox(
mailing_list = mbox_mailing_list,
# start_year_month = mbox_start_year_month,
save_folder_path = mbox_save_folder_path,
verbose = TRUE
)