Skip to content

build: perform parallel builds of BlocksRuntime #395

New issue

Have a question about this project? Sign up for a free GitHub account to open an issue and contact its maintainers and the community.

By clicking “Sign up for GitHub”, you agree to our terms of service and privacy statement. We’ll occasionally send you account related emails.

Already on GitHub? Sign in to your account

Closed
wants to merge 1 commit into from
Closed
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
92 changes: 74 additions & 18 deletions CMakeLists.txt
Original file line number Diff line number Diff line change
Expand Up @@ -74,14 +74,12 @@ if(ENABLE_SWIFT)

set(INSTALL_TARGET_DIR "${INSTALL_LIBDIR}/swift/${SWIFT_OS}" CACHE PATH "Path where the libraries will be installed")
set(INSTALL_DISPATCH_HEADERS_DIR "${INSTALL_LIBDIR}/swift/dispatch" CACHE PATH "Path where the headers will be installed for libdispatch")
set(INSTALL_BLOCK_HEADERS_DIR "${INSTALL_LIBDIR}/swift/Block" CACHE PATH "Path where the headers will be installed for the blocks runtime")
set(INSTALL_OS_HEADERS_DIR "${INSTALL_LIBDIR}/swift/os" CACHE PATH "Path where the os/ headers will be installed")
endif()

if(NOT ENABLE_SWIFT)
set(INSTALL_TARGET_DIR "${INSTALL_LIBDIR}" CACHE PATH "Path where the libraries will be installed")
set(INSTALL_DISPATCH_HEADERS_DIR "include/dispatch" CACHE PATH "Path where the headers will be installed")
set(INSTALL_BLOCK_HEADERS_DIR "include" CACHE PATH "Path where the headers will be installed for the blocks runtime")
set(INSTALL_OS_HEADERS_DIR "include/os" CACHE PATH "Path where the headers will be installed")
endif()

Expand Down Expand Up @@ -132,35 +130,93 @@ endif()

option(INSTALL_PRIVATE_HEADERS "installs private headers in the same location as the public ones" OFF)

find_package(BlocksRuntime QUIET)
if(NOT BlocksRuntime_FOUND)
if(NOT CMAKE_SYSTEM_NAME STREQUAL Darwin)
set(BlocksRuntime_INCLUDE_DIR ${PROJECT_SOURCE_DIR}/src/BlocksRuntime)

add_library(BlocksRuntime
STATIC
${PROJECT_SOURCE_DIR}/src/BlocksRuntime/data.c
${PROJECT_SOURCE_DIR}/src/BlocksRuntime/runtime.c)
set_target_properties(BlocksRuntime
set(LIBDISPATCH_BLOCKSRUNTIME_CMAKE_ARGS
-DCMAKE_AR=${CMAKE_AR}
-DCMAKE_BUILD_TYPE=${CMAKE_BUILD_TYPE}
-DCMAKE_C_COMPILER=${CMAKE_C_COMPILER}
-DCMAKE_C_FLAGS=${CMAKE_C_FLAGS}
-DCMAKE_INSTALL_PREFIX=<INSTALL_DIR>
-DCMAKE_MAKE_PROGRAM=${CMAKE_MAKE_PROGRAM}
-DCMAKE_LIBRARY_PATH=${CMAKE_LIBRARY_PATH}
-DCMAKE_RANLIB=${CMAKE_RANLIB}
-DCMAKE_TOOLCHAIN_FILE=${CMAKE_TOOLCHAIN_FILE}
-DINSTALL_PRIVATE_HEADERS=${INSTALL_PRIVATE_HEADERS})

include(ExternalProject)
ExternalProject_Add(SharedBlocksRuntime
SOURCE_DIR
${PROJECT_SOURCE_DIR}/src/BlocksRuntime
CMAKE_ARGS
${LIBDISPATCH_BLOCKSRUNTIME_CMAKE_ARGS}
-DBUILD_SHARED_LIBS=YES
INSTALL_COMMAND
${CMAKE_COMMAND} -E env --unset=DESTDIR ${CMAKE_COMMAND} --build . --target install
BUILD_BYPRODUCTS
<INSTALL_DIR>/lib/${CMAKE_SHARED_LIBRARY_PREFIX}BlocksRuntime${CMAKE_SHARED_LIBRARY_SUFFIX}
<INSTALL_DIR>/lib/${CMAKE_IMPORT_LIBRARY_PREFIX}BlocksRuntime${CMAKE_IMPORT_LIBRARY_SUFFIX}
BUILD_ALWAYS
1)
Copy link

@kevints kevints Oct 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

My experience using ExternalProject hasn't been great compared to creating library targets that are directly addressable in the build system and I strongly suggest we don't use it in dispatch.

  1. We need to propagate all possible interesting CMake variables from the outer build system to the inner ExternalProject build system via CMAKE_ARGS. For example here you don't propagate CMAKE_C_FLAGS, CMAKE_AR or CMAKE_LIBRARY_PATH which could cause the downstream libraries to be built differently. There are too many influential variables to enumerate here.
  2. ExternalProjects don't get good dependency tracking compared to targets that cmake understands natively. You have to declare BUILD_BYPRODUCTS manually and it's easy to under or over-depend. ExternalProjects also don't have great IDE integration as they appear opaque.
  3. Setting up a library target as IMPORTED prevents dependency tracking and prevents it being exported from the build system (via install(EXPORT) for example).
  4. Ninja is not designed to be invoked recursively - you can get into a situation where the downstream ninja is performing tasks equal to the number of CPUs, so you can have (N^2-1) tasks running in parallel (where N is the number of cores your build machine has). Console output is also broken.

Having said all that, have you considered making BlocksRuntime and BlocksRuntime_static targets using add_library instead?

Copy link
Member Author

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Well, we already are using ExternalProject for libdispatch in swift, and it has been working quite well :-). That said, there is interest in moving to the LLVM wrapper. Doing so will automatically propagate those other variables implicitly.

The dependency tracking issue that you portray is not entirely correct. There is tracking of the dependencies in the sense that the external project's byproducts are linked into the imported library target. So if the sub-library changes, CMake/ninja realize that and rebuild the dependent targets.

Yes, I did consider that. I suppose that if we want to just duplicate the rules, thats certainly an option. I really am strongly opposed to creating the custom wrappers for both variants. I suppose that we could always extract the BlocksRuntime rules into a separate CMakeList and include it in this one to make this easier to work with.

ExternalProject_Add(StaticBlocksRuntime
SOURCE_DIR
${PROJECT_SOURCE_DIR}/src/BlocksRuntime
CMAKE_ARGS
${LIBDISPATCH_BLOCKSRUNTIME_CMAKE_ARGS}
-DBUILD_SHARED_LIBS=NO
INSTALL_COMMAND
${CMAKE_COMMAND} -E env --unset=DESTDIR ${CMAKE_COMMAND} --build . --target install
BUILD_BYPRODUCTS
<INSTALL_DIR>/lib/${CMAKE_STATIC_LIBRARY_PREFIX}BlocksRuntime${CMAKE_STATIC_LIBRARY_SUFFIX}
BUILD_ALWAYS
1)

ExternalProject_Get_Property(SharedBlocksRuntime install_dir)
add_library(BlocksRuntime::BlocksRuntime SHARED IMPORTED)
set_target_properties(BlocksRuntime::BlocksRuntime
PROPERTIES
POSITION_INDEPENDENT_CODE TRUE)
if(HAVE_OBJC AND CMAKE_DL_LIBS)
set_target_properties(BlocksRuntime
PROPERTIES
INTERFACE_LINK_LIBRARIES ${CMAKE_DL_LIBS})
INTERFACE_INCLUDE_DIRECTORIES
${PROJECT_SOURCE_DIR}/src/BlocksRuntime
IMPORTED_LOCATION
${install_dir}/lib/${CMAKE_SHARED_LIBRARY_PREFIX}BlocksRuntime${CMAKE_SHARED_LIBRARY_SUFFIX}
IMPORTED_IMPLIB
${install_dir}/lib/${CMAKE_IMPORT_LIBRARY_PREFIX}BlocksRuntime${CMAKE_IMPORT_LIBRARY_SUFFIX})

if(ENABLE_SWIFT)
set(LIBDISPATCH_BLOCKSRUNTIME_HEADER_DIR ${SWIFT_LIBDIR}/swift/Block)
set(LIBDISPATCH_BLOCKSRUNTIME_LIBDIR ${SWIFT_LIBDIR}/swift/${SWIFT_OS}/${SWIFT_HOST_ARCH})
set(LIBDISPATCH_BLOCKSRUNTIME_STATIC_LIBDIR ${SWIFT_LIBDIR}/swift_static/${SWIFT_OS}/${SWIFT_HOST_ARCH})
else()
set(LIBDISPATCH_BLOCKSRUNTIME_HEADER_DIR ${CMAKE_INSTALL_INCLUDEDIR})
set(LIBDISPATCH_BLOCKSRUNTIME_LIBDIR ${CMAKE_INSTALL_LIBDIR})
set(LIBDISPATCH_BLOCKSRUNTIME_STATIC_LIBDIR ${CMAKE_INSTALL_LIBDIR})
endif()

add_library(BlocksRuntime::BlocksRuntime ALIAS BlocksRuntime)

install(FILES
${PROJECT_SOURCE_DIR}/src/BlocksRuntime/Block.h
DESTINATION
"${INSTALL_BLOCK_HEADERS_DIR}")
${LIBDISPATCH_BLOCKSRUNTIME_HEADER_DIR})
if(INSTALL_PRIVATE_HEADERS)
install(FILES
${PROJECT_SOURCE_DIR}/src/BlocksRuntime/Block_private.h
DESTINATION
"${INSTALL_BLOCK_HEADERS_DIR}")
${LIBDISPATCH_BLOCKSRUNTIME_HEADER_DIR})
endif()
install(FILES
${install_dir}/lib/${CMAKE_SHARED_LIBRARY_PREFIX}BlocksRuntime${CMAKE_SHARED_LIBRARY_SUFFIX}
DESTINATION
${LIBDISPATCH_BLOCKSRUNTIME_LIBDIR})
if(CMAKE_SYSTEM_NAME STREQUAL Windows)
install(FILES
${install_dir}/lib/${CMAKE_IMPORT_LIBRARY_PREFIX}BlocksRuntime${CMAKE_IMPORT_LIBRARY_SUFFIX}
DESTINATION
${LIBDISPATCH_BLOCKSRUNTIME_LIBDIR})
endif()
install(FILES
${install_dir}/lib/${CMAKE_STATIC_LIBRARY_PREFIX}BlocksRuntime${CMAKE_STATIC_LIBRARY_SUFFIX}
DESTINATION
${LIBDISPATCH_BLOCKSRUNTIME_STATIC_LIBDIR})
endif()

check_symbol_exists(__GNU_LIBRARY__ "features.h" _GNU_SOURCE)
Expand Down
51 changes: 51 additions & 0 deletions src/BlocksRuntime/CMakeLists.txt
Original file line number Diff line number Diff line change
@@ -0,0 +1,51 @@

cmake_minimum_required(VERSION 3.4.3)
project(BlocksRuntime
LANGUAGES C)

include(CheckIncludeFiles)
include(GNUInstallDirs)

if(CMAKE_C_SIMULATE_ID STREQUAL MSVC)
# clang-cl interprets paths starting with /U as macro undefines, so we need to
# put a -- before the input file path to force it to be treated as a path.
string(REPLACE "-c <SOURCE>" "-c -- <SOURCE>" CMAKE_C_COMPILE_OBJECT "${CMAKE_C_COMPILE_OBJECT}")
endif()

set(CMAKE_C_STANDARD 11)
set(CMAKE_C_STANDARD_REQUIRED YES)

set(CMAKE_C_VISIBILITY_PRESET hidden)
set(CMAKE_C_VISIBILITY_INLINES_HIDDEN YES)

check_include_files("objc/objc-internal.h" HAVE_OBJC)

option(INSTALL_PRIVATE_HEADERS "install private headers in the same location as the public ones" NO)
Copy link

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

By placing this inside an ExternalProject there's no way to trigger this option anymore.

Copy link
Member Author

@compnerd compnerd Oct 3, 2018

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Oh, but there is! There is the still the option in libdispatch, which just passes it through :-) However, note that the installation for the swift side is done in the libdispatch side, so that we can install both variants into the install'ed image for libdispatch.


add_library(BlocksRuntime
${PROJECT_SOURCE_DIR}/data.c
${PROJECT_SOURCE_DIR}/runtime.c)
set_target_properties(BlocksRuntime
PROPERTIES
POSITION_INDEPENDENT_CODE TRUE)
if(HAVE_OBJC AND CMAKE_DL_LIBS)
target_link_libraries(BlocksRuntime
PUBLIC
${CMAKE_DL_LIBS})
endif()

install(FILES
${PROJECT_SOURCE_DIR}/Block.h
DESTINATION
${CMAKE_INSTALL_INCLUDEDIR})
if(INSTALL_PRIVATE_HEADERS)
install(FILES
${PROJECT_SOURCE_DIR}/Block_private.h
DESTINATION
${CMAKE_INSTALL_INCLUDEDIR})
endif()
install(TARGETS
BlocksRuntime
DESTINATION
${CMAKE_INSTALL_LIBDIR})